Models do not read long inputs evenly. A research paper found that performance is often best when relevant information sits at the beginning or end of the input, and drops when it is in the middle, even for models built for long contexts. So retrieving more is not always better.
What the paper found
The paper "Lost in the Middle: How Language Models Use Long Contexts" tested how well models use information depending on where it appears in the input. Its conclusion was that where the relevant content sits strongly affects how well it gets used. Performance was often highest at the beginning or end, and fell noticeably in the middle. The authors concluded that current models do not reliably use information throughout long inputs.
Models and context sizes have changed since the paper was published, so do not assume the exact effect holds for the model you use today. Treat it as a reason to test, not as a law.
What it means for retrieval
When you build a search backed question answering tool, you pick how many chunks to pass to the model. Two forces pull in opposite directions:
- Too few, and the answer may not be in the set at all.
- Too many, and the answer may be present but buried.
Anthropic reports that in their experiments, passing 20 chunks worked better than 5 or 10. That is one data point from their datasets. Your best number depends on your chunk size, your model and your questions.
What to do about it
- Rank well, then place well. Put the strongest chunks first, and consider placing the second strongest last, so the weaker ones fall in the middle.
- Rerank. Retrieve a wide candidate set, then use a reranker to pick a small set of the best. See reranking explained.
- Trim chunks. Shorter, focused chunks put less filler between facts.
- Test several values of k. Try 5, 10 and 20 and compare answers on a fixed question set.
- Ask for citations. If the model must name the file for each claim, you can spot answers that ignored the right chunk.
Do not skip retrieval for a giant prompt
If the whole corpus fits in the window, you can skip retrieval, but you inherit the same position effect across all of it. See when to put the whole knowledge base in the prompt.
A small experiment you can run
Take one question with a known answer in one chunk. Build prompts with that chunk at position 1, in the middle, and last, padded with unrelated chunks. Ask the same question each time. If correctness varies, you have measured the effect on your model, cheaply, in an afternoon.
Quick check
Does a larger context window solve this?
Not by itself. The paper found the effect even in models designed for long contexts. A larger window lets you include more, but it does not guarantee even use.
Is the effect the same for every model?
Unknown. Test yours.
Sources
Last reviewed: 19 October 2026.
