If your knowledge base is small enough to fit in the model's context window with room to spare, you may not need retrieval at all. Anthropic's guidance suggests that under about 200,000 tokens, roughly 500 pages, you can include everything in the prompt.
The simplest system that could work
Retrieval pipelines have parts: chunking, embedding, an index, a ranking step, and a freshness job. Each part can fail quietly. If your material is a style guide, a runbook, an API reference or one small service's code, skipping all of that is a real option.
Anthropic's article on contextual retrieval states the rule directly: if the knowledge base is under 200,000 tokens, about 500 pages, include the whole thing in the prompt and no retrieval is needed. They add that prompt caching can make this faster and cheaper, citing latency reduced by more than 2 times and costs cut by up to 90 percent. Those are their figures for their platform. Check the current documentation for whichever model you use, as limits and prices change.
How to estimate your size
Rough method: count characters in the files you would include and divide by four to estimate tokens. It is crude, and code often tokenises less efficiently than prose, so leave a wide margin. Then use the provider's token counter for a real number.
A mid sized microservice may fit. A monorepo will not.
Why bigger prompts are not free
Two problems appear as you fill the window.
Cost and speed. Every question pays to process the whole prompt unless it is cached.
Attention. Research on long contexts found that models use information best when it is at the beginning or end of the input, and worse when it is in the middle. See why more context does not always help. A prompt stuffed with everything can answer worse than a prompt with the right ten chunks.
A decision table
- Under the context limit, rarely changes, few users: Whole prompt.
- Under the limit, many questions a day: Whole prompt with caching, if available.
- Over the limit: Retrieval.
- Per user access rules: Retrieval with permission filters.
- Needs citations to specific files: Either, but retrieval makes it easier.
A middle path
Include a compact map in every prompt, such as a file tree and one line per module, then retrieve full text for the few modules that matter. This keeps the model oriented without paying for the full corpus each time.
Test it
Take twenty real questions. Answer them with whole prompt, and with a simple retrieval setup. Compare accuracy and cost. If the simple approach wins, ship it and revisit when the corpus grows. For a method, see how to evaluate a code search system.
Quick check
Is a bigger context window the end of RAG?
No. Windows are finite, cost scales with prompt size, and attention is uneven within long inputs. Retrieval still matters for large or permissioned corpora.
What happens when the corpus grows past the limit?
Your answers degrade or the request fails. Set an alert on corpus size so you move to retrieval before that happens.
Sources
Last reviewed: 19 October 2026.
