Distillation trains a small student model to reproduce the behaviour of a larger teacher model or an ensemble. You keep most of the useful behaviour at a fraction of the size, which makes serving cheaper and faster. It works best on narrow tasks.
Where the idea comes from
The widely cited paper is Geoffrey Hinton, Oriol Vinyals and Jeff Dean's 2015 work, Distilling the Knowledge in a Neural Network. Its starting point was practical. Averaging the predictions of many models usually improves results, but running a big ensemble is too expensive to deploy. The authors compress the ensemble's knowledge into a single model that is much easier to deploy.
They reported strong results on MNIST and showed that distilling an ensemble into one model significantly improved a heavily used commercial acoustic model.
How it works in practice
The student does not just learn from hard labels. It learns from the teacher's output distribution, which carries more information than a single right answer. A teacher that says a picture is 70 percent cat and 25 percent dog tells the student something a bare "cat" label does not.
For language models, a common modern pattern is simpler. You run the teacher over a large set of prompts, collect its responses, filter them, and fine-tune the student on those pairs. That is closer to synthetic data generation than to the original method. The risks are covered in synthetic training data: uses and risks.
What to check before you distill
- Task scope. A student can match a teacher on a defined task, such as classifying support tickets. It is unlikely to match it across everything.
- Terms of use. Providers set rules on using model outputs to train other models. Read the current terms for the teacher you plan to use. We did not verify any provider's terms for this post.
- Teacher errors. The student inherits the teacher's mistakes and may amplify them.
- Eval coverage. Test the student on rare and hard cases, not only typical ones.
Distillation versus other ways to shrink a model
| Approach | What changes | Typical use |
|---|---|---|
| Distillation | New small model learns from a big one | Narrow task at low cost |
| Quantization | Same model, lower numeric precision | Fit a model on less memory |
| Fine-tuning a small base | Train an existing small model on your data | Format, style, domain |
These combine well. You can distill into a small model, then quantize it for serving. Our explainer on quantization covers the second step.
A sensible workflow
- Define the task and an eval set with real inputs.
- Measure the teacher on that eval set.
- Generate and filter teacher outputs for training.
- Train the student and compare it with the teacher.
- Ship only if the gap is acceptable for the use case.
Quick check
Is distillation the same as fine-tuning?
They overlap. Distillation is fine-tuning where the training signal comes from a teacher model rather than from human labels.
Will the student be as good as the teacher?
Not in general. It can come close on a narrow task. Measure it on your own evals.
Sources
- Distilling the Knowledge in a Neural Network, fetched 19 October 2026.
Last reviewed: 19 October 2026.
