Speculative decoding explained: faster local inference, same output

Speculative decoding uses a small, fast model to guess several tokens ahead, then has the large model check all the guesses in one pass. Accepted guesses are free speed, and the output distribution stays the same as the large model alone. The gain depends on how often the small model guesses right.

The idea in plain terms

A large model generates one token per pass, so producing K tokens takes K serial runs. The original paper by Leviathan, Kalman and Matias, "Fast Inference from Transformers via Speculative Decoding", starts from two observations. Some tokens are easy enough for a much smaller model to predict, and a large model can check several proposed tokens in parallel.

So the method runs like this:

1. The draft model proposes a short run of tokens.
2. The target model scores that whole run in one pass.
3. Tokens that match what the target would have chosen are accepted. At the first mismatch, the target's own token is used and the process repeats.

The paper says this speeds up sampling without changing the output, and that it works with existing off the shelf models without retraining or architecture changes. On T5-XXL they report a speedup of roughly 2 to 3 times over the standard implementation with identical outputs.

Why the output stays the same

The accepted tokens are exactly those the large model would have produced, per the paper's guarantee on the output distribution. The draft model only affects speed. A poor draft model is slow, not wrong.

When it helps

Gains depend on the acceptance rate. Predictable text, such as code boilerplate or structured output, tends to accept more draft tokens. Creative or high entropy text accepts fewer. The paper's numbers are for one model family and setup, so measure your own.

It also costs memory, because the draft model must be loaded next to the large one. On a tight VRAM budget that can matter more than the speed gain. See how much VRAM a model needs.

Support in local tools

The vLLM docs list speculative decoding methods including n-gram, suffix, EAGLE and DFlash. The llama.cpp README page I fetched did not mention speculative decoding, so check its current docs and release notes for your build. I did not find a speculative decoding setting in the Ollama FAQ either.

How to evaluate it for your setup

1. Choose a draft model from the same family as your main model, if the tool requires matching vocabularies. Check the tool's docs.
2. Measure tokens per second with and without it on the same prompts.
3. Compare outputs at temperature zero for a sample. They should match if the implementation is exact.
4. Weigh the speedup against the extra memory.

If the speed gain is small on your tasks, skip it and spend the memory on a larger context or a higher quant. See choosing a quant level.

Quick check

Sources

• Fast Inference from Transformers via Speculative Decoding (arXiv)
• vLLM documentation
• llama.cpp README

Last reviewed: 19 October 2026.