The Chinchilla paper found that for a fixed compute budget, model size and the number of training tokens should grow in equal proportion. Doubling parameters means doubling tokens. It argued many large models had been undertrained, which helps explain why small models trained on lots of data can be strong.
What the paper says
The abstract states that for compute optimal training, the model size and the number of training tokens should be scaled equally. The authors argued that existing large models had grown in size while keeping training data roughly fixed, leaving them undertrained.
That finding changed how people plan training. Instead of just making a model bigger, you ask how many tokens it should see.
Why it matters for small models
If a larger model is undertrained, a smaller model trained on more tokens can match it at lower serving cost. Serving cost is paid on every request, so for a model used heavily a smaller, longer trained model can be the cheaper choice overall.
The Phi-3 report fits this picture. It describes phi-3-mini at 3.8 billion parameters trained on 3.3 trillion tokens, with a heavily filtered data mix. See small language models: when a small model is enough.
What it does not tell you
- Quality of data. Chinchilla is about quantity relative to size. Phi and LIMA style results point to data quality as a separate lever. See how much data to fine-tune.
- Fine-tuning. The result concerns pretraining. Tuning an existing model on a few thousand examples follows different rules.
- Your task. A compute optimal model is not automatically the best for your narrow job.
- Inference cost. The paper optimizes training compute. If you will serve a model a lot, going past the training optimum can still be sensible. We did not verify specific ratios from the paper beyond the equal scaling statement.
Practical takeaways for a team
- You rarely pretrain from scratch. You choose among released models, so look at both parameter count and the tokens it was trained on, when published.
- A smaller model with more training data is a candidate worth testing, not a guaranteed win.
- Fine-tuning data needs are small. Spend effort on quality and evals.
- Compare options on your own eval set, including cost per thousand requests.
Quick check
What does Chinchilla say in one line?
For compute optimal training, model size and training tokens should be scaled equally.
Does it apply to fine-tuning?
No. It addresses pretraining compute. Fine-tuning data needs depend on your task and base model.
Sources
- Training Compute-Optimal Large Language Models, fetched 19 October 2026.
- Phi-3 Technical Report, fetched 19 October 2026.
Last reviewed: 19 October 2026.
