Late interaction: one vector per token

1.Three ways a query can meet a passage

A single-vector model, the kind every earlier lesson used, squeezes each chunk into one point and the query into another. The two never meet. The cross-encoder from Reranking reads query and passage together, which is accurate and cannot be precomputed. Late interaction sits between them. It encodes query and passage separately, but keeps one small vector for every token instead of pooling them. The query meets the passage only at scoring time, and the meeting is cheap: arithmetic on vectors that already exist.

The score is called MaxSim, from the 2020 ColBERT paper by Khattab and Zaharia. For each query token, find the passage token it matches best and keep that one cosine. Then add the kept values up. Take the query "laptop return deadline" and the passage "Electronics may be returned within 30 days". Pretend a model has turned each token into a 2-d vector (real ones have about 128 numbers; two lets you check with a pencil). Every number below is invented.

Now score the desk passage from Reranking, "Our returns desk is open on weekdays", the same way. It has one perfect match, 1.00 for "return" against "returns". But nothing in it is close to "laptop" or "deadline", whose best cells are 0.59 and 0.57. The total is 2.16 against 2.85. One strong word cannot carry a passage, because every part of the question has to find something.

A single vector averages a passage into one point. Late interaction lets each word of the question go and find its own evidence.

2.Where it helps, and what it costs

It tends to help on long passages where one paragraph decides relevance, on questions with several requirements, and on data unlike what a dense model was tuned on. The evidence is mixed. A Hugging Face post from August 2026 (vendor-reported) compares a matched pair from LightOn: a dense model and a late-interaction model with the same size, backbone and training data. On thirteen small datasets the late-interaction model averaged about one nDCG point higher and lost on four, including FiQA2018, a finance question set. A long-document test in the same post showed a far wider gap in its favour. So: competitive on average, clearly better on long documents, not better everywhere.

What it costs

At index time you pay one encoder pass per passage, as with a dense model, plus storage. At query time you pay one pass for the query, then a grid of dot products per candidate: 50 candidates, 125 vectors each, a 12-token query is 50 x 12 x 125 = 75,000 dot products.

Storage is the bill that surprises people. In the same post, one small corpus came out at about 125 token vectors per passage, about 42 times the size of a 384-dimension dense model on the same passages, and several times even the largest dense model it tried. One corpus, one machine, vendor-reported. Scale it yourself. Ten million chunks, 125 vectors each, 128 numbers per vector, 4 bytes per number is 640 GB. One 1,024-number dense vector per chunk is about 41 GB, close to the size worked out in Vector databases.

Three levers shrink the bill. Compress each vector: ColBERTv2 (2022) stores a nearest centroid plus a small residual, which its authors report shrinks the index several times over. Pool tokens: merge each passage's most similar vectors. The token-pooling paper (September 2024) reports about half the vectors with almost no loss. Or fix the count: ConstBERT (ECIR 2025) encodes every passage as the same number of vectors whatever its length.

Speed at scale is the gap. PLAID (2022, from the ColBERT authors, tested at over a hundred million passages) and MUVERA (2024, NeurIPS, from Google, which approximates the score with one vector per passage) report their own numbers. A search in October 2026 found no independent latency or recall figure at over a hundred million passages. One independent study of PLAID at MS MARCO scale (2024) found its settings finely balanced.

3.Three ways to use it, and when to skip it

As of October 2026 the Qdrant docs (multivectors, from v1.10) and the Weaviate docs (from v1.29) describe MaxSim support. The Hugging Face post says Elasticsearch and OpenSearch can rescore with MaxSim but cannot retrieve with it. Check your own store. In a full stack it sits between the first stage and an optional cross-encoder, narrowing about 1,000 candidates to 100 (an illustration from Pinecone's May 2025 post, not a measurement).

Start with rescoring. The same post calls it "considerably cheaper per candidate" than a cross-encoder but gives no latency figure, so measure it. The code below ran on Sentence Transformers v6.1 (as of October 2026), with the small teaching model all-MiniLM-L6-v2 (as of October 2026) as stage one and LightOn's LateOn (as of August 2026) as stage two.

# pip install -U sentence-transformers
import os
from sentence_transformers import MultiVectorEncoder, SentenceTransformer, util

# Set both to current checkpoints. Passing similarity_fn_name="meanmaxsim"
# to MultiVectorEncoder gives scores on a bounded scale.
dense = SentenceTransformer(os.environ["DENSE_MODEL"])
late = MultiVectorEncoder(os.environ["LATE_MODEL"])

passages = [
    "Our returns desk is open on weekdays from 9am until 5pm.",
    "Electronics may be returned within 30 days of delivery.",
    "Laptop batteries are covered by a two year warranty.",
    "Clothing can be exchanged in store with a receipt.",
    "Gift cards are not refundable and never expire.",
    "Opened software is not eligible for return.",
]
query = "how long do I have to return a laptop"

# Stage one: the single-vector search you already have.
doc_vecs = dense.encode_document(passages, convert_to_tensor=True)
q_vec = dense.encode_query(query, convert_to_tensor=True)
hits = util.semantic_search(q_vec, doc_vecs, top_k=4)[0]
shortlist = [passages[h["corpus_id"]] for h in hits]

# Stage two: token vectors for the shortlist only, thrown away afterwards.
scores = late.similarity(late.encode_query([query]), late.encode_document(shortlist))[0]
for i in scores.argsort(descending=True).tolist():
    print(f"{scores[i].item():7.3f}  {shortlist[i]}")

#  11.091  Electronics may be returned within 30 days of delivery.
#  11.030  Laptop batteries are covered by a two year warranty.
#  10.930  Opened software is not eligible for return.
#  10.870  Our returns desk is open on weekdays from 9am until 5pm.

The comments show the output of one run. The order matched the dense order, and the four scores sit within a quarter of a point. That bunching is normal, and it feeds the first failure.

The score that looks like a similarity. MaxSim adds one cosine per query token, so the total grows with the query. The same post saw it across models: one query scored about 11 on one model and 12.8 to 27 on ColBERTv2, which pads queries to 32 tokens. So never set a cutoff from one model's scores, never compare scores across queries, and never mix two models in one index. Divide by the query length, or rank within one query only.

The call that fails silently. These models are asymmetric, hence separate query and document calls. The ColBERT-Zero paper (February 2026, the authors' own runs) is about training: a prompt mismatch between pre-training and fine-tuning cost about 1.3 points on BEIR. The same logic suggests that encoding a query with the document call will hurt too. I did not measure it, but never do it. And the quiet one: a 5,000-passage pilot stores a few hundred megabytes, so nobody sees the 640 GB coming.

When not to use it. Adopt it after hybrid search and a reranker, and only if your eval set from Measuring retrieval shows the right chunk is missing or badly ordered. Skip it for a corpus of a few thousand chunks, where a cross-encoder is cheap and already good; when a strong reranker has closed the gap; when your store cannot do MaxSim and the corpus is too big to rescore; and when your data resembles the sets where the dense model won. Do not train one: the ColBERT-Zero authors report hundreds of GPU-hours for the full recipe, so use released checkpoints.

The same design appears for page images, where a model keeps a vector per image patch instead of per token. Retrieving pages as images covers when that beats parsing.