Use supervised fine-tuning (SFT) when you have correct example outputs, direct preference optimization (DPO) when you have pairs of better and worse outputs, and reinforcement fine-tuning when you can grade answers automatically or with expert judges, usually for reasoning tasks.
Supervised fine-tuning
SFT trains on inputs paired with correct responses. OpenAI's model optimization guide says it suits classification, nuanced translation and specific output formats.
Data needed: input and ideal output. Effort is in writing or collecting good examples. It is the simplest method and the right default when you know what the answer should be.
Direct preference optimization
DPO trains on pairs: a preferred response and a rejected one. The OpenAI guide says it suits things like summarization focus and tone.
The DPO paper explains why it became popular. Standard RLHF first fits a separate reward model on human preference labels, then uses reinforcement learning to tune the model against it. The authors call that pipeline complex and often unstable. DPO reparameterizes the problem so it reduces to a simple classification loss, needing no sampling from the language model during tuning and no significant hyperparameter tuning. They report it matched or beat existing methods on sentiment control, summarization and single-turn dialogue.
Data needed: prompts with chosen and rejected answers. These can be easier to collect than perfect answers, because it is often easier to say which of two is better.
Reinforcement fine-tuning
Reinforcement fine-tuning grades generated responses and reinforces higher scoring reasoning. The OpenAI guide says it applies only to reasoning models and needs expert graders who agree on the ideal outputs.
Data needed: prompts plus a grader. The grader is the hard part. If experts disagree, the signal is noise.
Side by side
| Method | You provide | Good for | Main risk |
|---|---|---|---|
| SFT | Ideal outputs | Format, classification, style | Labels teach bad habits |
| DPO | Better and worse pairs | Tone, focus, preference | Weak or inconsistent preferences |
| Reinforcement | A grader | Reasoning with checkable answers | Gaming the grader |
How to choose
- If you can write the right answer, start with SFT.
- If you can only rank answers, try DPO, often after an SFT step.
- If answers can be checked by a program or consistent experts, consider reinforcement fine-tuning.
- Measure each against the same held-out evals. See writing evals before you fine-tune.
For AI generated preference labels instead of human ones, read RLAIF versus RLHF explained.
Check availability
Which methods you can run depends on the provider or the open source library. The OpenAI guide says the platform is being wound down for new users, so confirm current availability before planning.
Quick check
Which method needs the least setup?
SFT. You need example inputs and ideal outputs and nothing else.
Why is the grader so important in reinforcement fine-tuning?
The model learns whatever the grader rewards. A vague or inconsistent grader teaches the wrong thing.
Sources
- OpenAI model optimization guide, fetched 19 October 2026.
- Direct Preference Optimization, fetched 19 October 2026.
Last reviewed: 19 October 2026.
