What Is a Vector Database?

Specialised infrastructure for approximate nearest-neighbour search, trading exactness for speed. Under about a hundred thousand vectors, brute force or pgvector will serve you better than a new service to operate.

Reviewed

A vector database stores embeddings and answers one question quickly: which stored vectors are closest to this one?

That is a narrower job than database suggests, and the narrowness is the point. The interesting engineering is entirely in making nearest-neighbour search fast at scale.

The problem it solves

Finding the nearest vectors is trivial in principle: compute the distance to every stored vector and take the top handful. This is exact, simple, and perfectly adequate for a surprising amount of real work.

It also scales linearly. A hundred million vectors of a thousand dimensions each means a hundred billion multiply-adds per query. At that point you need an index, and every vector index makes the same trade: give up the guarantee of finding the true nearest neighbours in exchange for finding almost all of them, far faster. This is approximate nearest neighbour search, and approximate is not a hedge — it is the design.

The dominant algorithm is HNSW, from Yu. A. Malkov and D. A. Yashunin's Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs (2016). It builds a layered proximity graph and navigates it from coarse to fine, achieving logarithmic complexity scaling — which is why almost every vector store on the market implements it or a variant.

The knobs you get are all versions of the same trade: higher recall costs latency and memory. A system returning 95% of the true nearest neighbours is usually indistinguishable in output quality from one returning 100%, at a fraction of the cost.

What you actually need beyond similarity

Pure similarity search is rarely enough in production, and the features that separate the options are mostly these:

Metadata filtering. Nearest chunks, but only from documents this user may see, in English, updated this year. This is the single most important capability and the one implementations differ on most. Filtering after the vector search can return nothing when the filter is selective; filtering during the search is harder to build and much better. If your application has any access control, this is your primary evaluation criterion — and it is a genuine security boundary, not a convenience. See prompt injection for why retrieved content is a trust boundary too.

Hybrid search. Combining keyword and vector scoring. In practice this beats pure vector search often enough that its absence is a real limitation.

Updates and deletes. Some indexes handle churn gracefully; others degrade until rebuilt. If your corpus changes hourly this matters enormously, and if you index a static document set once it does not matter at all.

Operational reality. Backups, replication, whether it can be the thing that pages you at 3am.

You may well not need one

This is the part the category's marketing does not lead with.

Under roughly a hundred thousand vectors, brute force in memory is fine. A NumPy dot product over a hundred thousand embeddings is a few milliseconds. You get exact results, no index to tune, no service to run, and no recall to reason about. A great many production RAG systems are comfortably in this range and have added a distributed vector database to solve a problem they do not have.

If you already run Postgres, try pgvector first. Your embeddings live next to the rows they describe, metadata filtering is just SQL with a real query planner, and transactions, backups and access control already exist and are already understood by your team. The operational saving is substantial and the ceiling is higher than people expect. Several other established databases have added vector types on the same logic.

A dedicated vector database earns its place when you have tens of millions of vectors or more, when query volume is high enough that latency at recall is a genuine engineering constraint, when you need the corpus sharded across machines, or when you want the surrounding machinery — hybrid search, reranking, managed embedding pipelines — as a product rather than as something you assemble.

The honest ordering is: brute force, then the database you already run, then a specialised one when you can name the specific limit you hit. Adding infrastructure is easy and removing it is not.

How to choose, if you do need one

Evaluate on your own corpus rather than on published benchmarks, because benchmark datasets have different dimensionality, clustering and filter selectivity from yours, and all three change the answer.

Measure recall at your latency budget, not either in isolation — any system can be fast at low recall or accurate when slow. Measure it with your filters applied, because filtered performance is where implementations diverge sharply and where published numbers are least representative. Then check the cost at your actual vector count and query rate, including the memory: HNSW indexes are typically held in RAM, and that is usually the dominant line.

Finally, note that switching costs are lower than they feel. Your embeddings are just arrays; re-indexing is mechanical. Changing your embedding model is the expensive migration, because it forces re-embedding the entire corpus — see what are embeddings.

Where it sits in a system

The database is one component and rarely the one limiting quality. In a typical retrieval pipeline — chunk, embed, index, retrieve, rerank, generate — the vector store handles a single step, and disappointing results usually trace to chunking or to retrieval strategy rather than to the index.

RAG vs tools vs long context covers whether you need retrieval at all; RAG vs fine-tuning vs prompt engineering covers whether retrieval is the right fix for your failure; and what is a context window explains why retrieving less and better usually beats retrieving more.

The short version

A vector database is specialised infrastructure for approximate nearest-neighbour search, trading exactness for speed via indexes like HNSW. The features that matter in practice are metadata filtering, hybrid search and update handling. And the first question to ask is whether your corpus is large enough to need one — under about a hundred thousand vectors, brute force or pgvector will serve you better than a new service to operate.

More in the AI development pack.

Sources

← Back to Articles How we research & review →