A reranker takes the top candidates from a fast first search and rescores each one by reading the query and the passage together. It is slower per item than comparing precomputed vectors, so it runs on a short list. It often improves which results reach the model.
Two kinds of model
Embedding models (also called bi-encoders) turn the query and each document into vectors separately. You compute document vectors once, ahead of time, and compare at query time. That makes search over a large index fast.
Cross-encoders take the query and one passage together and output a relevance score. The Sentence Transformers documentation describes them as models that calculate similarity scores directly for pairs of texts, with a rank method that orders a list of candidates for a query. They cannot be precomputed, since every query changes the pair, so scoring a whole corpus would be slow.
The two stage pattern
- Fast stage: keyword and vector search returns, say, 100 to 150 candidates.
- Slow stage: the reranker scores each candidate against the query and keeps the best 10 to 20.
The Sentence Transformers page describes this as typical: embed the corpus and the query, find the closest matches, and often refine the top results with a reranker.
Anthropic's retrieval article follows the same shape. They retrieved about 150 candidate chunks, scored them with a reranker, and passed the top 20 to the model. They note reranking adds a small amount of latency. Their reported failure rate, measured as one minus recall at 20, was 2.9 percent for contextual embeddings plus contextual keyword search, and 1.9 percent with reranking added. That is their data, not a promise for yours.
Why it helps
A bi-encoder squeezes a whole chunk into one vector before it sees the question. A cross-encoder sees both, so it can notice that a chunk mentions the exact condition the user asked about. It is better at fine distinctions among already plausible results.
Why it may not help
- If the right chunk is not in the candidates, no reranker can find it. Recall of the first stage sets the ceiling.
- Added latency matters for interactive tools.
- A generic reranker may not understand code or your domain. Test it.
- It adds another model to run, update and secure.
When to add one
Add a reranker when your measured results show the right chunk is often in the top 50 but not the top 5. That pattern means ordering is the problem. If the right chunk is missing from the top 50, fix chunking, add keyword search or change the embedding model first. See chunking source code.
Interaction with context position
Reranking lets you pass fewer, better chunks, which reduces the burying problem described in lost in the middle.
Quick check
Can a reranker run locally?
Cross-encoder models are available as open models and run through libraries such as Sentence Transformers. Check size and speed on your hardware.
Should I rerank every query?
If latency is acceptable, yes. Otherwise rerank only when the first stage scores are close.
Sources
Last reviewed: 19 October 2026.
