For style, format and narrow tasks, often a few hundred to a few thousand clean, varied examples are enough. Quality and coverage matter more than raw count. The LIMA paper tuned a 65 billion parameter model on only 1,000 curated prompt and response pairs.
What LIMA showed
The LIMA paper took a 65B LLaMa model and fine-tuned it with standard supervised training on 1,000 curated pairs, with no reinforcement learning and no human preference modeling. The authors conclude that almost all knowledge in large language models is learned during pretraining, and that instruction tuning mainly teaches the model to produce high-quality output with little data.
That is a claim about teaching style and format on a strong base model. It does not mean a thousand examples will teach new facts or a hard skill.
A rough sizing ladder
These are working rules of thumb, not figures from a paper. Adjust them against your evals.
| Goal | Starting size | Notes |
|---|---|---|
| Output format or tone | 50 to 500 | Variety matters more than volume |
| Classification with a few labels | 200 to 2,000 | Balance the classes |
| Extraction into a schema | 300 to 3,000 | Include messy and edge inputs |
| New domain behaviour | Thousands | Check if retrieval fits better |
How to find your number
- Hold out an eval set first and never train on it.
- Train on 100 examples and score.
- Double the data, retrain, and score again.
- Stop when gains flatten. That is your point of diminishing returns.
This learning curve costs a few small runs and saves you from labelling thousands of examples you did not need. If runs are expensive, use an adapter method such as LoRA.
What beats more data
- Cleaner labels. One wrong example in twenty can teach the wrong habit.
- Coverage. Include rare cases, refusals, empty inputs and long inputs.
- Consistency. Two annotators who disagree produce a confused model. Write a short labelling guide.
- Deduplication. Near copies inflate counts without adding information.
Our fine-tuning data preparation checklist goes through these steps.
When you really do need a lot
Teaching a model a new language, a specialist notation or a skill far outside its pretraining takes much more data, and often continued pretraining rather than simple instruction tuning. At that point, compare the cost against using a larger base model with good prompts.
Quick check
Is 1,000 examples always enough?
No. LIMA shows it can work for instruction style on a strong 65B base model. Your task, base model and quality bar decide.
Should I add more data or fix the data I have?
Fix first. Review errors from your eval set and trace them to bad or missing examples.
Sources
- LIMA: Less Is More for Alignment, fetched 19 October 2026.
Last reviewed: 19 October 2026.
