Choosing an embedding model

There are a lot of embedding models and the good ones are close together. That is the honest starting position. The gap between the tenth-best model for your use case and the best one is usually smaller than the gap between good chunking and bad chunking. So spend your effort proportionally — pick sensibly, measure once, and go and fix your chunks.

Names and numbers in this space go stale in months. What follows is the shape of the decision, which does not.

1Five axes, in the order they bite

Quality on your domain. Not quality in general. A model trained mostly on web prose can be noticeably worse on dense contract language, medical notes, or code. This is the axis you have to measure yourself, and section 2 is how.

Dimensions. How many floats per vector. This is the one that follows you around, because it sets storage and search cost for the life of the system. Ten million chunks at 1536 dimensions in 4-byte floats is about 60 GB of raw vectors before any index overhead; the same corpus at 384 dimensions is about 15 GB. That is the difference between fitting in memory and not.

Max input length. The token ceiling per call. If your chunks are 800 tokens you have room almost everywhere. If you want to embed whole documents, this becomes the binding constraint, and a model that silently truncates instead of erroring will lose the back half of every long chunk without telling you.

Cost per million tokens. Hosted embedding pricing sits in the low cents to low tens of cents per million tokens, which is one to two orders of magnitude cheaper than generation. Look it up on the vendor's pricing page before you commit — do not trust a number from a blog post, including this one. The number that surprises people is not the indexing cost, which is a one-off, but the query cost at high traffic, which is not.

Can it run on your hardware. Open models via sentence-transformers run on a CPU for small corpora and on a modest GPU for large ones. That matters when your data cannot leave the building, when you want no per-query network hop, or when you need to re-embed 50 million chunks and would rather not pay an API for it.

The usual sensible shortlist: a hosted model from Voyage, OpenAI, Cohere or Google when you want quality without operating anything, and an open model from the sentence-transformers ecosystem when you want control or cannot send data out. Both ends of that are fine choices. Which specific model leads is a question with a three-month shelf life, so check the leaderboard rather than a tutorial.

2Measure it on your own queries

The MTEB leaderboard is a genuinely useful filter. It tells you which models are in the running and which are not. It cannot tell you which one is best for you, because it is an average over dozens of tasks that are not your task, and the top models are separated by fractions of a point.

Here is the whole method, and it is smaller than you think.

Write down 30 real questions — from your support inbox, your logs, your own head if that is all you have. For each one, find the chunk that should come back and note its id. That labelled set is an hour of work and it is the most valuable artefact in your retrieval project. Then measure recall@k: the fraction of queries where the right chunk appears in the top k results. Run it once per candidate model.

# pip install numpy
import numpy as np

# Your corpus, and thirty real queries with the chunk that should win.
chunk_ids   = ["doc-14#2", "doc-14#3", "doc-31#0", ...]
chunk_texts = ["Use Forgot password on the sign-in page...", ...]
labelled    = [("how do I reset my password", "doc-14#2"), ...]   # 30 of these

def recall_at_k(embed, k=5):
    """embed(texts, kind) -> list of unit-length vectors. One per candidate model."""
    index = np.array(embed(chunk_texts, kind="document"))
    hits = 0
    for question, want in labelled:
        q = np.array(embed([question], kind="query")[0])
        top = np.argsort(index @ q)[::-1][:k]
        hits += want in {chunk_ids[i] for i in top}
    return hits / len(labelled)

for name, embed in candidates.items():        # candidates: {"model-a": fn, "model-b": fn}
    print(f"{name:24} recall@5 = {recall_at_k(embed):.2f}")

Read the result like an engineer. If two models are within a couple of points on 30 queries, that is noise — pick the cheaper or faster one and stop thinking about it. If everything scores below about 0.6, your problem is not the model. It is your chunking, and lesson 3 is where you should be.

Model choice is a shortlist from the leaderboard and a decision from your own data. Anyone who picked purely from a leaderboard has not measured their system.

3The costs you are signing up for

Dimensions are a lever, not a fixed cost. Some models are trained with Matryoshka representation learning, which pushes most of the signal into the leading dimensions. That means you can keep the first 512 of a 1536-dimension vector, renormalise it, and still have a usable embedding — a third of the storage for a small, measurable quality drop. Measure the drop on your labelled set with the same recall_at_k above. If losing a point of recall saves you two thirds of a large index, that is often the right trade. Truncating a model not trained this way just breaks it, so check the model card first.

Changing the model is a migration. This is the constraint that catches teams six months in. Vectors from different models are not comparable, so switching means re-embedding every chunk and rebuilding every index. On a large corpus that is hours of compute, a real bill, and a window where old and new vectors must not mix. Plan for it the way you plan a schema change: version the model name in your chunk records, build the new index alongside the old one, cut over reads atomically, then drop the old one. If you cannot name which model produced a given vector in your database, you cannot do that safely — so store it.

Prefixes and modes. Many retrieval models want to know whether the text is a query or a document, either through an input_type parameter or a literal prefix string you prepend. The rules are per-model and they are in the model card.

All three fail silently. There is a cheap defence: put the embedding call behind one function that takes kind="query" or kind="document" and refuses to run without it, exactly as in the snippet above. Nobody in your codebase should ever call the raw client directly.