When to fine-tune an LLM and when better prompting is enough

Start with prompting and examples, and fine-tune only when evals show that is not enough. Fine-tuning pays off for fixed output formats, shorter prompts, tone, or a smaller model doing one narrow task. It is a poor way to teach a model new facts that change often.

Try the cheap levers first

Prompt engineering means giving the model clear instructions, relevant context and a few worked examples. OpenAI's model optimization guide says plainly that prompt engineering may be enough for your use case. It is also fast to change. You edit text, rerun a test set, and see the result in minutes.

Fine-tuning, by contrast, needs a dataset, a training run and a new model version to manage. The same guide notes it can be a time-consuming process.

Cases where fine-tuning helps

The guide lists several situations where training makes sense:

• You need more examples than fit in a single context window.
• You want shorter prompts to save tokens and latency.
• You want to train on sensitive data without including it in every request.
• You want a smaller model that does one specific task well.

Format and style are the classic wins. If you need strict JSON in a house schema, or a support reply in a consistent voice, a few hundred good examples can teach that more reliably than a long instruction block.

Cases where it does not help

Fine-tuning is a weak tool for facts that change. A product catalogue, a policy document or yesterday's incident report belongs in retrieval, where you can update a document without retraining. If your problem is "the model does not know our data," look at retrieval first. We compare the two in fine-tuning versus RAG as a decision guide.

It also will not fix a task you cannot measure. If nobody can say what a good answer looks like, training data will encode that confusion.

A simple decision order

1. Write an eval set from real inputs and set a baseline.
2. Improve the prompt and add examples. Rerun the evals.
3. Add retrieval if the gap is missing knowledge.
4. Fine-tune if the gap is behaviour, format, cost or latency.
5. Rerun the evals and compare the tuned model against the best prompt.

The model optimization guide describes the same loop of evals, prompting and fine-tuning, repeated as a flywheel. Our post on writing evals before you fine-tune covers step one in detail.

Check platform availability

Provider tooling changes. The OpenAI guide says the company is winding down its fine-tuning platform, with new users unable to access it at the time we fetched the page. Before you plan around a hosted fine-tuning service, check the provider's current documentation. Open weight models trained with your own tooling are the other route.

Quick check

Sources

• OpenAI model optimization guide, fetched 19 October 2026.

Last reviewed: 19 October 2026.