Synthetic data, meaning examples generated by a model, can help you bootstrap a fine-tuning set cheaply. It works best when filtered, mixed with real data and checked against real evals. Used carelessly it copies the generator's errors and can narrow the model.
What worked in published research
The Self-Instruct paper had a pretrained model generate instructions, inputs and outputs, filtered out invalid or near duplicate samples, then fine-tuned the original model on what remained. Applied to vanilla GPT-3, the authors report a 33 percent absolute improvement on Super-NaturalInstructions, roughly matching InstructGPT-001, which used private user data and human annotation.
The phi-1 paper trained a 1.3 billion parameter code model on about 6 billion tokens of textbook quality web data plus about 1 billion tokens of synthetic textbooks and exercises generated with GPT-3.5. It reports 50.6 percent pass@1 on HumanEval and 55.5 percent on MBPP, with 4 days of training on 8 A100 GPUs.
Both show a pattern: generated data helped when it was filtered or curated for quality, not dumped in raw.
What can go wrong
A separate paper, The Curse of Recursion, reports that training on model generated content causes lasting damage. The rare, low probability parts of the original data distribution disappear, an effect the authors call model collapse. They observed it in variational autoencoders, Gaussian mixture models and LLMs. We look at how worried a normal team should be in model collapse: myth versus fact.
Other practical risks:
- Inherited errors. The generator's mistakes become your labels.
- Narrow style. Generated text tends to sound alike, so the tuned model does too.
- Terms of use. Providers may restrict training on their outputs. Check the current terms. We did not verify any in this post.
- Eval contamination. Generated examples can echo public benchmarks.
A safer recipe
- Start with a seed set of real, high quality examples.
- Generate variations with a strong model and a clear prompt.
- Filter hard: validity checks, deduplication, rules, and sampled human review.
- Blend synthetic with real data, and keep real data in the eval set.
- Compare against a model trained on real data only.
- Track which rows are synthetic so you can remove them later.
Where it fits best
- Formats and schemas where correctness can be checked by a program, such as valid JSON or code that passes tests.
- Rare cases that real logs hardly contain, such as unusual error messages.
- Bootstrapping before you have real traffic.
Where to avoid it
- Facts, since a generator can invent them confidently.
- Anything where real human behaviour is the target.
- Evaluation. Never evaluate on data produced by the model family you are testing.
Quick check
Is synthetic data always bad?
No. Published work shows gains when it is filtered and curated. Problems arise from raw, unchecked or purely recursive use.
Should evals use synthetic data?
Prefer real inputs. If you must generate, have humans review them and keep them separate from training.
Sources
- Self-Instruct: Aligning Language Models with Self-Generated Instructions, fetched 19 October 2026.
- Textbooks Are All You Need, fetched 19 October 2026.
- The Curse of Recursion: Training on Generated Data Makes Models Forget, fetched 19 October 2026.
Last reviewed: 19 October 2026.
