Use retrieval (RAG) when the model needs facts that change or must be cited. Use fine-tuning when you need consistent behaviour, format, tone, or a smaller model for one task. Many production systems use both: retrieval for knowledge, tuning for how the answer is written.
What each one changes
Retrieval leaves the model alone. At question time it fetches relevant documents and puts them in the prompt. Update a document and the next answer reflects it.
Fine-tuning changes the model's weights. It teaches patterns from examples. It is not a reliable way to store specific facts, and you cannot easily remove one fact later.
The OpenAI model optimization guide lists the cases where tuning helps: more examples than fit in a context window, shorter prompts, training on sensitive data without sending it in every request, and a smaller model that performs a specific task well. Notice that none of those is "teach it our latest documents."
Questions to ask
- Does the knowledge change? If weekly or faster, retrieve.
- Do answers need sources? Retrieval can point at the passage it used. A tuned model cannot show where a fact came from.
- Is the problem format or tone? Tune.
- Are prompts too long and costly? Tuning may let you drop long instructions and examples.
- Do you need to delete or restrict a fact? Retrieval lets you remove a document. Removing something from weights is hard.
- Is the model too big for the job? A tuned small model may be cheaper, see small language models.
Comparison
| Factor | Retrieval | Fine-tuning |
|---|---|---|
| Fresh facts | Strong | Weak |
| Citations | Possible | Not available |
| Output format | Prompt dependent | Strong |
| Upfront effort | Index and chunk | Data and training |
| Per request cost | Longer prompts | Shorter prompts |
| Access control | Filter at retrieval time | Hard once trained in |
Using both
A common pattern: retrieve passages, then pass them to a model tuned to answer in a fixed structure, cite the passages, and decline when the passages do not contain the answer. Tune on examples of exactly that behaviour, including refusals.
Security notes
Retrieval has its own risks. Documents you retrieve may contain hidden instructions, and a poisoned index can steer answers. Training data can also be poisoned, and tuned models may leak memorised text. Neither is the safe option by default. Treat both data paths as untrusted input and test them.
Decide with evidence
Build an eval set before you commit, as in write evals before you fine-tune. Test a baseline prompt, then retrieval, then tuning, and compare.
Quick check
Can fine-tuning replace a knowledge base?
No. It can absorb some facts, but unreliably, with no citations and no easy updates. Keep documents in a retrievable store.
Do I have to pick one?
No. Retrieval for knowledge plus tuning for behaviour is a common combination.
Sources
- OpenAI model optimization guide, fetched 19 October 2026.
Last reviewed: 19 October 2026.
