How to evaluate a code search system with a 30 question test set

Write 30 real questions, record the file or function that answers each, and measure how often it appears in the top 5 and top 20 results. This is recall at k. It takes a few hours, and it turns arguments about chunking and models into numbers.

Why bother

Retrieval systems fail quietly. A bad index still returns results, and the language model writes a fluent answer from whatever it got. Without a test set, you find problems when a developer complains. With one, you can change a setting and see whether it helped.

Step 1: collect questions

Use real ones. Mine your team chat, support threads and onboarding notes for questions like "where do we retry failed webhooks?" or "what sets the session timeout?" Aim for variety: some with exact identifiers, some described in plain words, some that span two files.

Thirty is enough to see big differences. It is not enough to see small ones, so do not over read a change of one question.

Step 2: record the answers

For each question, write down the file path or function that should be retrieved. If two places are acceptable, list both. Have a second person check a sample, since your own knowledge biases you.

Store it as a simple file:

{"q": "where do we verify webhook signatures", "expected": ["src/hooks/verify.py"]}

Step 3: define recall at k

Recall at k, for one question, is whether an expected item appears in the first k results. Average across questions. Anthropic's retrieval article reports a failure rate defined as one minus recall at 20, which is a useful number to track: it counts the questions where the right chunk never reached the model.

Track recall at 5 and at 20. Recall at 5 shows how good the ranking is. Recall at 20 shows whether the answer is available at all.

Step 4: run your baseline

Start with keyword search, see the SQLite FTS5 how to. Record the scores and keep the file in version control.

Step 5: change one thing at a time

Compare, one at a time:

• chunking by function versus fixed size
• keyword only, vector only and hybrid, see hybrid retrieval
• two candidate embedding models, see choosing an embedding model
• with and without a reranker
• number of chunks passed on

Change one variable, rerun, record.

Step 6: read the failures

The score tells you how much, the failures tell you why. For each miss, see whether the expected chunk was ranked 6th, or absent entirely. If absent, check whether it was ever indexed.

Step 7: also test answers

Retrieval is half the system. For a handful of questions, read the final answers and check them against the code. Ask for file citations so you can verify.

Keep it honest

Do not tune on the same 30 questions forever. Add new ones from real usage, and hold some back as a check. Never put private code or personal data in a test set that is shared outside the team.

Quick check

Sources

• Contextual Retrieval, Anthropic

Last reviewed: 19 October 2026.