How to Choose a Vector Database for RAG

Choose a vector database by sizing your actual retrieval workload: chunk count, peak query rate, filter complexity, update frequency, and latency budget. A vector database that wins benchmarks can still lose in production if your filters are selective or your corpus churns daily. For most teams under a few million chunks, pgvector inside the Postgres you already run is the confident starting point; managed services like Pinecone earn their price at high scale and high QPS. This post gives you six workload questions and a fully worked sizing example.

Why Your Retrieval Workload Should Pick Your Vector Database

Here is how this decision usually goes wrong. A team reads a benchmark post, falls for the engine with the prettiest latency chart, ships it - and three months later learns the real problem was a chunking strategy that split tables mid-row, or a tenant filter that turned every query into a crawl. The store was never the bottleneck. They just chose it before they understood what they were asking it to do.

The workload lens fixes this. Five properties decide the right store long before any vendor comparison does: how many chunks you actually have, how many queries arrive at peak, how selective your filters are, how often the corpus changes, and how many milliseconds the rest of the pipeline leaves for retrieval. A 200,000-document internal knowledge base serving 5 queries per second has almost nothing in common with a 50-million-chunk product catalog powering live chat, and pretending a single recommendation covers both is how bad architecture gets written.

So here is the promise: six questions you can answer in an afternoon, one fully worked sizing example you can copy with your own numbers, and an honest verdict on the pgvector versus Pinecone debate that surfaces in every planning meeting.

What Actually Matters in Vector Search Performance?

Before the framework, it helps to know what the marketing pages are actually arguing about. TechTarget's vector database overview grounds the taxonomy nicely; what follows is the practitioner's version.

Recall, latency, and the ANN tradeoff

Every serious engine runs approximate nearest neighbor search, because exact search over millions of high-dimensional vectors is too slow to ship. HNSW builds a layered graph you can hop through in a handful of jumps; IVF sorts vectors into buckets and searches only the promising ones. Both give up a little recall to buy a lot of speed, and both expose knobs that slide along that curve. The practical takeaway: recall and latency are one dial with two labels, and any benchmark quoting one without the other is telling you half the story.

Metadata filtering: the benchmark killer

Production RAG almost never runs a naked similarity search. There is always a tenant ID, a document ACL, a date range. Engines handle this very differently - some filter before the ANN pass, some after - and once your filter matches only 2% of the corpus, the gap between those two behaviors can stretch to an order of magnitude.

The true cost of embeddings storage

Embeddings storage is a multiplication problem: dimensions times 4 bytes times chunks times replicas, with index overhead piled on top. Each factor looks innocent on its own. Multiplied together, they routinely ambush teams who priced only the happy path - especially when a multi-embedding strategy doubles the chunk count overnight. The worked example below runs this arithmetic in full.

The Six-Question Workload Framework

Answer these six questions honestly and the vendor decision mostly makes itself. They map straight onto the ingestion-to-answer flow we walk through in how to build a RAG pipeline, where retrieval is one stage among several.

1. How many chunks, really? Count after chunking, after overlap, after any multi-embedding strategy. Documents are a vanity metric; chunks are what the index actually holds, and the ratio between the two is often 10 to 1 or worse.
2. What is your query rate at peak? Averages hide the Monday-morning spike that takes the demo down. Size for the burst, then check what the burst costs.
3. How selective are your filters? Tenant isolation, per-document permissions, and date ranges each narrow the candidate set in different ways. High selectivity is the single most common reason a chosen engine disappoints.
4. How often does the corpus change? An append-only archive and a knowledge base re-embedded nightly are effectively different products. Frequent updates punish engines with expensive index rebuilds.
5. What latency budget is left for retrieval? The LLM eats most of your response time. If generation takes 3 seconds, the difference between 40ms and 120ms of retrieval is invisible to users, which changes what fast enough means.
6. Who operates this? A platform team with an on-call rotation can run anything. Two engineers who also own the API, the frontend, and the deploy pipeline should buy operations instead of tuning HNSW parameters at 2am.

Notice that none of these questions mention a vendor. That is deliberate. The answers are facts about your system, and facts are far harder to argue with than benchmarks.

Pinecone vs pgvector: When the Boring Answer Wins

Now for the question everyone actually came for, and the answer that disappoints anyone hunting for drama: both are good, and your workload answers pick between them.

Where pgvector comfortably wins

pgvector rides the Postgres you already run, so it inherits the transactions, backups, permissions, and monitoring you already trust. Vectors live next to the source rows, which means a document update and its embedding refresh commit atomically, and your metadata filters are plain SQL. For corpora into the low millions of chunks at moderate query rates, anything else is genuinely hard to justify: one fewer vendor, one fewer network hop, one fewer thing to explain in the architecture review.

Where a managed vector database earns its price

Pinecone-style managed services earn their keep in three situations: tens of millions of vectors, high sustained QPS under strict latency SLAs, and teams who would rather index tuning be somebody else's job. That last one matters more than engineers like to admit. If nobody on your team wants to own ANN index operations, paying a vendor to own them is a rational engineering decision, and there is no shame in it.

The rest of the field in one honest paragraph

Beyond those two poles sit purpose-built engines like Milvus and Weaviate, search platforms like Elasticsearch and OpenSearch that bolted vectors onto mature keyword infrastructure, and cloud-native options like Azure AI Search, whose architecture Microsoft covers well on its tech community. Built In's ecosystem coverage maps the wider landscape if you want the full catalog. The short version: search platforms shine when you need hybrid keyword-plus-vector search anyway, and cloud-native options shine when you are already deep in that cloud.

Read the table as a conversation starter with your own answers, because a single strained cell that matches your hardest constraint outweighs five strong ones that don't.

A Worked Sizing Example: 40,000 Documents to a Number

Let's make this concrete with a scenario I see constantly: a B2B company with 40,000 pages of documentation and blog content, building an assistant for customers and support staff.

Step 1: from documents to chunk count

Chunk at roughly 500 tokens with a 15% overlap and an average page yields about 10 chunks, putting us near 400,000 chunks in total. The first lesson lands already: the chunking decision multiplied the corpus by 10, which is a bigger lever than anything on a vendor pricing page.

Step 2: from chunks to storage and memory

With 1536-dimension embeddings at 4 bytes per float, each vector costs about 6 KB. Multiply by 400,000 chunks and you get roughly 2.4 GB of raw vectors; add HNSW index overhead plus a replica and a realistic budget is 5 to 7 GB, which fits comfortably in memory on a modest Postgres instance.

This is exactly the kind of arithmetic worth doing precisely instead of by feel. Inside AstroFabric, our pipeline agent runs sizing math like this in a code sandbox, so the numbers behind a recommendation are computed rather than estimated, and we would push any team to hold its own planning to the same standard.

Step 3: from workload answers to a decision

Layer in the rest: 20 QPS at peak, per-tenant filters, a weekly content refresh. Every answer points the same way. The corpus is small, the query rate modest, the filters plain SQL, and weekly updates trivial for a transactional store. The verdict here is pgvector, confidently - and here is exactly what would flip it: chunk count growing past a few million, peak QPS climbing toward the hundreds, or the team deciding it wants zero index operations. Any one of those moves the recommendation to a managed service, and you will see it coming quarters in advance.

How Does the Vector Store Fit the Rest of the RAG Pipeline?

Zoom out and the store shrinks. In the full Retrieval-Augmented Generation loop - ingestion, chunking, embedding, retrieval, reranking, generation - the vector store is one stage, and in my experience it is rarely the weakest one.

Hybrid search beats vendor switching

If answer quality disappoints, the highest-leverage fix is usually hybrid search: dense vector similarity paired with old-fashioned keyword matching. Dense retrieval misses exact part numbers, error codes, and proper nouns; keyword search catches them. Adding a keyword leg typically improves retrieval quality more than any engine migration ever will.

Reranking: the cheapest quality upgrade

The second cheapest fix is a reranker. Retrieve 50 candidates fast, then let a cross-encoder reorder the top handful with far more care than any ANN index can afford. It costs milliseconds, buys precision, and works identically no matter which store you chose. Retrieval quality is also a large part of why the RAG vs fine-tuning debate usually resolves in retrieval's favor for fresh, changing knowledge: good retrieval keeps answers current without retraining anything.

When retrieval becomes a decision

The frontier is agentic RAG, where an agent decides when to retrieve, what to query, and whether the results justify another pass. That pattern raises query volume unpredictably, which matters when you answer question two of the framework: an agent that retrieves three times per user turn just tripled your QPS estimate.

Your Decision Checklist and Migration Escape Hatch

Here is the whole framework compressed into something you can run today.

And here is the pressure release valve: embeddings are portable. They are arrays of floats, and every store ingests them. Keep the embedding step decoupled from storage behind a thin retrieval interface, and a wrong first pick costs a re-index job over a weekend instead of an architecture rewrite. That reversibility should make you bolder about starting simple.

So the practitioner stance: start with the store your team can operate confidently, measure retrieval quality weekly against a fixed evaluation set, and upgrade when the numbers tell you to. There is a bonus waiting at the end of that discipline. Content your own pipeline retrieves cleanly is structured, chunked, and answerable - exactly the content AI assistants quote when they cite sources. Good retrieval hygiene pays twice.

Put Your Sizing on Solid Ground

If you would rather have this kind of analysis computed than guessed, AstroFabric's specialist agents run exact calculations in a code sandbox, propose changes behind approval gates, and meet you in the console, over REST or MCP, or right inside Slack. Sign up and put real numbers behind your next infrastructure decision.

Frequently asked questions

Sources

• TechTarget: Vector database overview
• Microsoft Tech Community: Vector search architecture
• Built In: Vector database ecosystem coverage