Vector databases

Part 1 gave you vectors. Now you need somewhere to keep them, and the industry has a strong opinion about what that should be. The opinion is often premature.

Here is the reframe that makes the decision easy: a vector database is not a new kind of maths. The similarity scoring is the same dot product everywhere. What differs is the index that avoids doing it a million times, the filtering that runs alongside it, and how much of your week it takes to operate.

1What you are actually buying

Strip the marketing off and a vector store is four things stacked together.

Rows one and three you can do yourself. Rows two and four are the actual product. Notice that neither of them matters at small scale: with 10,000 vectors you do not need an index, and with a corpus you rebuild from source in five minutes you do not need durability.

That is why the honest advice is a ladder, not a recommendation.

2The ladder

Rung one: a numpy array. Load your vectors into one (n, d) array, score with a matrix multiply, take the top k with argsort. This is exact — it compares against every vector, so it is perfectly accurate by construction — and it is fast up to roughly tens of thousands of vectors. It has no service, no client library, no network hop, and you can print the scores when something looks wrong, which is worth more during development than any feature on this page. Pickle the array beside a list of chunk texts and you have persistence.

Stop being embarrassed about this rung. Most internal tools never need to leave it.

Rung two: pgvector. If your application data is already in Postgres — and for most teams it is — this is the natural next step. pgvector adds a vector column type and the index types from lesson 5. Vectors live in the same database as your users, documents and permissions, which means filters and joins against your own tables are just SQL, and an update to a document and its vector happens in one transaction. Nothing to operate that you were not already operating.

-- pgvector: schema, index, and a filtered nearest-neighbour query
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
  id         bigserial PRIMARY KEY,
  doc_id     text NOT NULL,
  tenant_id  text NOT NULL,
  updated_at timestamptz NOT NULL DEFAULT now(),
  body       text NOT NULL,
  embedding  vector(1024) NOT NULL          -- must match your model's dimensions
);

CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON chunks (tenant_id);

-- <=> is cosine distance, so smaller is closer. Order by it and take the top 5.
SELECT id, doc_id, body, 1 - (embedding <=> $1) AS similarity
FROM chunks
WHERE tenant_id = $2
ORDER BY embedding <=> $1
LIMIT 5;

Note vector(1024) — the dimension is fixed in the schema. Changing embedding model means an ALTER TABLE and a full re-embed, which is exactly the migration lesson 2 warned about, now written down in DDL.

Rung three: a dedicated store. Qdrant, Weaviate, Milvus and Chroma self-hosted; Pinecone and the managed offerings from the others if you would rather not run it. You climb here for reasons you can state: tens of millions of vectors, sustained write throughput, filtering behaviour Postgres struggles with, sharding, or a team that wants the ops handled.

3Choosing between them, and when to move

Read the memory column implicitly: index structures like HNSW want your vectors in RAM. Ten million vectors at 1024 dimensions is around 40 GB of floats before index overhead, and that number — not query volume — is usually what forces a move.

The signals that you have genuinely outgrown a rung are specific, and none of them are "it feels small-scale":

• Search latency at your real corpus size exceeds your budget, measured, not guessed.
• Vectors no longer fit comfortably in the memory of the machine you have.
• You need concurrent writes and deletes without rebuilding an index.
• You need filtering that your current rung does badly — lesson 6 is where that gets specific.

Whatever rung you are on, keep the interface thin. One module exposing search(query, k, filters) -> list[Chunk] and one exposing upsert(chunks). Everything else in your codebase talks to those two functions and knows nothing about the store behind them. Then climbing a rung is one file, an afternoon, and a re-index — instead of a rewrite. This is the same discipline as putting the embedding call behind one function in lesson 2, and it pays off for the same reason: the parts of a retrieval system that change are the parts you should have wrapped.

Adding a vector database is an optimisation. Do it when a measurement tells you to, and pick the rung that matches your corpus rather than the one that matches your ambitions.