What an embedding really is

Most introductions describe an embedding as coordinates for meaning and leave it there. This course is the second pass: what the numbers are, why the distance measure is the one it is, and where the approach quietly stops working.

1.A fixed-length list of floats, and what it is not

An embedding model takes text and returns the same shape of output every time: a list of floating point numbers, a few hundred to a few thousand long. A word or a paragraph, same length out. That fixed length makes text something you can put in an array and compare with arithmetic. Every model also has an input limit. As of October 2026, the small teaching model used below, all-MiniLM-L6-v2, silently drops everything after about 256 word pieces (its model card says so).

The length is also your storage bill. That model returns 384 numbers of 4 bytes each, so 1,536 bytes per chunk, and a million chunks is about 1.5 GB before any index overhead. Double the dimensions and you double that.

Training shows the model pairs that should be close and pairs that should be far, and it adjusts until the geometry agrees. Here is what the resulting numbers are not:

That last row causes real outages. Re-embed half your corpus with a new model and search does not degrade gracefully - it returns confident nonsense from whichever half scores higher.

One vector per input is also the default, not the law. Some models keep a vector for every token, scored as in Late interaction: one vector per token. Some read the surrounding document first, as in Giving each chunk its context. Some embed a picture of a page, the subject of Retrieving pages as images. All still compare vectors by angle, so the rest of this lesson carries over.

An embedding is a position, and a position only means something relative to the other positions in the same space. There is no meaning inside one vector.

2.Cosine, and why not Euclidean

Picture each embedding as an arrow from the origin out to its point. Cosine similarity is the cosine of the angle between two arrows. Pointing the same way scores 1, at right angles 0, opposite -1. With the small model below, unrelated text lands near 0 and near-duplicates at 0.8 and up, a rough band for this one model, and another model puts the same pairs elsewhere. Read your own scores rather than trusting a threshold from a tutorial.

It is cosine rather than straight-line distance because in some models arrow length can track things you do not care about, like how long the text is, while angle tracks topic. Many embedding APIs also return unit-length vectors, already scaled so the arrow has length 1 (OpenAI's guide says its models do, as of October 2026; check yours). On unit vectors cosine is just the dot product, and Euclidean distance becomes a rearrangement of the same quantity, so it ranks results in exactly the same order. If vectors are not normalised, the two differ and Euclidean lets long texts drift on length alone: normalise first. The real bug is mixing them: build an index with cosine distance, query it with L2, and no error message tells you the results are wrong.

# pip install sentence-transformers numpy
import os
import numpy as np
from sentence_transformers import SentenceTransformer

# Set EMBED_MODEL to all-MiniLM-L6-v2 to reproduce the output below.
model = SentenceTransformer(os.environ["EMBED_MODEL"])

texts = [
    "How do I reset my password?",                                   # the query
    "Use Forgot password on the sign-in page to recover your login.",  # the answer
    "Invoices are charged to the card on file each month.",            # unrelated
    "How do I avoid resetting my password?",                           # the opposite
]
v = model.encode(texts, normalize_embeddings=True)   # unit length
q = v[0]

print("cosine    ", (v @ q).round(3))                  # dot product == cosine here
print("euclidean ", np.linalg.norm(v - q, axis=1).round(3))

# Two pairs used in section 3
for a, b in [("This ships to Canada", "This does not ship to Canada"),
             ("invoice INV-4471", "invoice INV-4472")]:
    x = model.encode([a, b], normalize_embeddings=True)
    print(round(float(x[0] @ x[1]), 2), "|", a, "|", b)

# Output from all-MiniLM-L6-v2 (384 dimensions), run in October 2026:
# cosine     [ 1.     0.584 -0.009  0.85 ]
# euclidean  [0.    0.912 1.421 0.547]
# 0.83 | This ships to Canada | This does not ship to Canada
# 0.99 | invoice INV-4471 | invoice INV-4472

Check one by hand. For the answer, d squared is 2 - 2 x 0.584 = 0.832, and its square root is 0.912. Both measures give the same order. Now read that order: the opposite question (0.85) outranks the sentence that actually answers (0.584). Nothing errored. That is the lesson of section 3.

3.What it is good at, and where it falls over

Embeddings are strong on the thing keyword search cannot do: matching meaning across different words. They are weak on the thing keyword search is trivially good at: exact strings.

Negation is the failure to remember, because it looks like success: a support bot answers "how do I reset my password" with the page on how to avoid resetting it, at a confident score. NevIR (Weller et al., EACL 2024) measured this by asking retrieval models to rank two passages that differ only by a negation. Most of the models it tested did no better than chance, and bi-encoders were among the worst. A better model narrows the gap; do not count on it to close it.

When not to rely on embeddings alone: when the query is an exact string, a number, a date, or a rule that must hold every time. The fixes are architectural. Keep a keyword index alongside for identifiers (Hybrid search), put dates and numbers in metadata columns you filter on (Metadata and filtering), and never expect a vector to enforce a rule.

One more asymmetry explains an API parameter you may have skipped. A question and the passage that answers it do not look alike: "What is the refund window?" shares little with "Requests submitted within 30 days of purchase are honoured in full." Retrieval models are trained on that mismatch, and many expose it as two modes, an input_type of query or document, or a prefix string you prepend. Embed documents in query mode, or queries in document mode, and recall drops with no error: everything still returns results, just quietly worse. Choosing an embedding model returns to this, because forgetting one half of the pair is a common bug in a working RAG system.

Every later lesson starts from this shape and these limits, beginning with Choosing an embedding model.