Do not pick an embedding model from a leaderboard alone. The MTEB authors found that no text embedding method dominates across all tasks. Shortlist two or three models that fit your hardware, run them on your own queries, and keep the one that retrieves the right files most often.
What the benchmark says
The MTEB paper evaluated 33 models across 8 embedding task types, 58 datasets and 112 languages. Earlier evaluations often used a few datasets from one task, so it was unclear how embeddings that excel at similarity would do at clustering or reranking. The authors concluded that no particular text embedding method dominates across all tasks. There is a public leaderboard and open source code.
The lesson is not "leaderboards are useless." It is that a high average hides weak spots, and your use, searching code, may not look like the average.
Constraints to settle first
- Where it runs. If code must stay on your machine, you need a local model. See running embeddings locally.
- Vector length. Longer vectors cost more storage and slower search. Ollama's documentation says length depends on the model, typically 384 to 1024 dimensions. If you use pgvector, note its indexing limits per vector type, such as 2,000 dimensions for the standard vector type.
- Input length. Check how many tokens a model accepts. A model that truncates your chunks quietly loses information.
- Language coverage. If your comments are not in English, check that the model handles your languages.
- Licence. Confirm the licence permits your use.
The comparison method
- Collect 30 to 50 real questions developers ask, such as "where do we verify webhook signatures."
- For each, note the files or functions that should come back.
- Index the same chunks with each candidate model.
- For each question, record whether the expected item appears in the top 5 and top 20.
- Compare the totals. Also look at the failures, not only the score.
The scoring approach is explained in how to evaluate a code search system.
Do not forget the baseline
Include keyword search in the comparison. For code, it sometimes beats a weak embedding model on identifier heavy questions, and it combines well with whichever model wins. See keyword or vector search.
Plan for change
You will change models eventually. Store the model name and chunk text alongside every vector, so a re-embed is a script rather than a project. Changing the model means rebuilding the whole index, since vectors from different models are incompatible.
Supply chain note
A model is a downloaded artefact. Pull from the original publisher, pin versions, and record where it came from. Our post on embedding model supply chain risks covers what to check.
Quick check
Is a bigger embedding model always better?
No. Bigger models cost more to run and store, and the benchmark finds no universal leader. Measure on your data.
How often should I re-evaluate?
When you change chunking, when new models appear that fit your hardware, or when your question set changes meaningfully.
Sources
Last reviewed: 19 October 2026.
