There is no universal answer. Anthropic reported that passing 20 chunks to the model worked better than 5 or 10 in their experiments, but chunk size, model and question type all change the result. Pick a starting value of 10 to 20, then measure on your own question set.
The trade-off
Retrieve too few chunks and the answer may not be among them. Retrieve too many and you pay for longer prompts, and the relevant text may sit where the model uses it poorly. Research on long inputs found that models often use information at the start and end best, and lose it in the middle. See lost in the middle.
What Anthropic reported
In their contextual retrieval write up, the authors state that they found passing 20 chunks to the model to be more effective than 5 or 10. Their pipeline retrieved about 150 candidates, reranked them and kept the top 20. Chunks there were generally a few hundred tokens at most. If your chunks are whole files, the same count means a much bigger prompt.
Factors that move the right number
- Chunk size. Smaller chunks need a larger k.
- Question type. A lookup of one function needs few chunks. "How does authentication work across services?" needs many.
- Reranker. A good reranker lets you pass fewer chunks.
- Model and context cost. Larger k means more tokens per question, so cost scales with it.
- Duplicates. Near duplicate chunks waste slots. Deduplicate.
Measure it in two steps
First, retrieval. With your test set from how to evaluate a code search system, compute recall at 5, 10 and 20. If recall at 20 is far above recall at 5, a small k throws away answers.
Second, generation. For a sample of questions, produce final answers at k of 5, 10 and 20, and grade them against the code. Recall measures availability. Grading measures use.
Often the curve rises, then flattens. Pick the smallest k near the flat part.
Adaptive k
Instead of a fixed number, set a score threshold and a maximum. Take chunks above the threshold, up to the cap. Easy questions get few chunks, hard ones get more. Keep a floor of, say, three, so the model always has something to work with.
Expand around hits
When a function chunk is retrieved, adding its neighbours, such as the class header or the callers, can help more than adding unrelated results. Keep a limit so expansion does not eat the budget.
Show your sources
Return file paths with each answer. If the answer cites a file that was not in the retrieved set, the model made it up, and you can catch that automatically. Whatever k you choose, log which chunks were sent, so failures can be debugged.
Quick check
Is a larger k always safer?
No. It raises cost and can bury the right chunk. Measure answer quality, not only retrieval.
Does k depend on the language model?
It can. Different models handle long inputs differently, so retest when you change the model.
Sources
Last reviewed: 19 October 2026.
