The KV cache holds the attention keys and values for every token the model has already seen, so it grows linearly with context length and with each parallel request. On a local machine it is the reason a model that loads fine can run out of memory when you paste a large file.
What the cache is for
A language model generates one token at a time. Each new token attends to all the earlier ones. Without a cache the model would recompute the keys and values for the whole history at every step. The Transformers documentation describes the KV cache as storing those calculations so they can be reused.
The cache is a good trade. It makes generation much faster. The cost is memory that scales with how much text is in play.
A size formula you can use
Hugging Face's optimization guide gives the count of stored values as two times the sequence length times the number of layers times the number of attention heads times the head dimension. For models that share key value heads across several query heads, use the number of key value heads instead of attention heads. Check the model's config file for the real numbers.
Here is a hypothetical model to show the scale. Assume 32 layers, 8 key value heads, head dimension 128, 16 bit values, and a 32,768 token context.
2 x 32768 x 32 x 8 x 128 = 2,147,483,648 values
2,147,483,648 values x 2 bytes = about 4.3 GB
That is one conversation. These numbers are invented for illustration, so use your model's config for a real estimate.
The guide makes a related point from its own measurements: for long inputs the cache can reach roughly half the size of the model weights on the model it tested.
Parallel requests multiply it
Ollama's FAQ says required RAM scales with the number of parallel requests times the context length. Its default for parallel requests is 1. If you raise it to serve a small team, the cache memory rises with it. Four parallel slots at the example context above would want about four times as much cache.
Ways to shrink it
- Use less context. Ollama's FAQ says its default context window is 4096 tokens. You often do not need more than that for short tasks.
- Quantize the cache. The same FAQ lists cache types: f16 by default, q8_0 at about half the memory, and q4_0 at about a quarter with more noticeable precision loss. It says the setting is global and applies when Flash Attention is enabled.
- Pick a model with grouped query attention. The Hugging Face guide explains that sharing key value heads shrinks the cache a great deal. Check the model card.
- Offload the cache. The Transformers docs describe moving cache layers to CPU memory at some throughput cost.
Test the loss, do not assume it
Cache quantization trades accuracy for memory, and the effect depends on the model and task. Run your own prompts at the lower setting before you adopt it. The guide to building a small eval shows a cheap way.
For the weight side of the budget, see how much VRAM a local model needs.
Quick check
Why does my model slow down or fail only on long prompts?
The weights are a fixed cost, but the KV cache grows with every token. A long prompt can push total memory past what you have, which forces swapping or an out of memory error.
Does quantizing the KV cache change the model files?
No. It changes how the runtime stores the cache while it runs. The weight file on disk is untouched.
Sources
- Optimizing LLMs for Speed and Memory, Hugging Face Transformers docs
- Cache strategies, Hugging Face Transformers docs
- Ollama FAQ
Last reviewed: 19 October 2026.
