A RAG pipeline is the production system that turns raw documents into grounded AI answers through six stages: ingestion, chunking, embedding, retrieval, reranking, and generation. Every stage of the rag pipeline has a distinct failure mode, and errors compound downstream, so a fluent final answer can hide a broken step three stages back. This walkthrough shows what each stage does, where it breaks in practice, and how to trace a real query end to end so you can find the weak link before your users do.
What Is a RAG Pipeline and Why Do Most of Them Fail Quietly?
Want the conceptual grounding first? Our explainer on Retrieval-Augmented Generation covers what RAG is and why it exists. This post is the part that piece deliberately leaves out: the operational machinery, stage by stage, and the specific ways each stage betrays you in production.
The six stages at a glance
Think of the pipeline as a relay race. Every runner trusts the baton they were handed. Nobody checks it mid-race.
| Stage | Purpose | Most common failure | Symptom users see | Fastest diagnostic |
|---|---|---|---|---|
| Ingest | Pull and parse source docs | Silent parse errors, stale docs | Confident answers from dead pages | Spot-check parsed text against originals |
| Chunk | Split docs into retrievable units | Structure shredded mid-table | Half-answers, orphaned pronouns | Read 20 random chunks aloud |
| Embed | Convert chunks to vectors | Domain mismatch, partial reindex | Jargon queries miss obvious docs | Query known doc with its own title |
| Retrieve | Find candidates fast | Top-k too small, ANN too aggressive | Right doc never surfaces | Check recall@20 vs recall@5 |
| Rerank | Reorder candidates precisely | Reranker trained off-domain | Relevant doc stuck at position 12 | Compare pre/post rerank ordering |
| Generate | Compose the grounded answer | Model ignores retrieved context | Plausible answers with no source | Ask for inline citations, verify them |
Why fluent output hides broken retrieval
Here is the trap. A broken calculator returns garbage you notice immediately. A broken RAG pipeline returns beautiful prose. I have watched a support bot quote a deprecated pricing page with perfect confidence, the formatting intact, a friendly line to close. Ingestion had added documents and never expired them, so the old page embedded just as convincingly as the new one. The user experience was flawless right up until the customer asked for the discount the bot promised.
Ingestion: Where Bad Data Enters and Never Leaves
Everything downstream inherits whatever ingestion produces. This is the highest-leverage stage, and still the one that gets skipped.
Parsing and cleaning real-world documents
Real corpora are ugly. PDFs mangle tables into single-column soup. HTML drags navigation bars and cookie banners into the extract. OCR turns "10%" into "1O%". The same policy document often lives in three slightly different versions across a wiki, a shared drive, and someone's export folder. Those near-duplicates are the nasty ones. They crowd retrieval with redundant chunks and shove genuinely different documents out of the top-k.
Metadata you will wish you had captured
Capture metadata at ingest time, because retrofitting it means re-crawling everything. At minimum:
- Source URL or file path, so answers can cite something clickable
- Publication and last-modified dates, so freshness can be scored
- Access permissions, so a sales rep never retrieves an HR document
- Document type and section hierarchy, for structure-aware chunking later
Build deletion in from day one. A pipeline that only adds documents drifts toward stale answers within months, quietly, one outdated policy at a time.
Document Chunking for RAG: The Decision That Shapes Everything Downstream
Chunking looks like a boring preprocessing detail. It is the decision that most shapes retrieval quality, because chunks are the atomic unit everything else operates on.
Fixed windows vs structure-aware splitting
The naive approach slices text every 500 tokens, indifferent to what the text is doing. Predictable wreckage follows. A pricing table splits mid-row, and the chunk that says "Enterprise tier" no longer sits with the chunk that holds its price. Retrieval finds one half. The generator improvises the other. You get an answer that references half a policy with total confidence. Structure-aware splitting keeps headings with their sections, honors list boundaries, and refuses to shred a table. It is worth every hour it takes to build.
Chunk size, overlap, and context windows
Small chunks embed precisely but strip context. Large chunks preserve context but dilute the embedding until nothing matches strongly. Most teams land somewhere between 200 and 500 tokens, with modest overlap so a sentence that straddles a boundary still survives. Two tricks punch above their weight. Prepend each chunk with its document title and section breadcrumb. And look at parent-child chunking, where you retrieve on the small chunks but hand the generator the larger parent section they came from.
Embeddings and Vector Search: Turning Text Into Retrievable Meaning
An embedding model compresses a chunk into a vector, and semantic similarity becomes geometric proximity. That is the whole trick. It works remarkably well inside the domain the model was trained on.
Choosing an embedding model without benchmarking theater
Feed dense legal text or clinical notes through a general-purpose model and recall degrades quietly. The model never learned that two pieces of jargon mean the same thing. You do not need weeks of benchmarking theater to catch this. Take fifty real queries, embed them with two or three candidate models, and count how often the known correct document lands in the top ten. Resources like IBM's material on retrieval architectures are useful for the landscape. Your own fifty queries still beat any public leaderboard. One operational rule is non-negotiable. If you ever swap embedding models, reindex everything. Vectors from different models live in incompatible spaces, and mixing them corrupts every similarity score.
Dense, sparse, and hybrid retrieval
Dense vectors are brilliant at meaning and terrible at exact identifiers. Ask for "error E4013" and semantic search happily returns documents about errors in general. Keyword search nails the identifier and then misses every paraphrase. Hybrid retrieval fuses dense and sparse scores so you get both. For a corpus full of SKUs, error codes, or part numbers, that fusion is the difference between a useful system and a frustrating one. On the index itself, stay with exact search until scale genuinely forces approximate nearest neighbors. When it does, tune ANN settings with recall measurements in hand rather than defaults on faith.
Why Does Retrieval Need a Second Opinion? Reranking Explained
First-stage retrieval is built for speed across millions of chunks, so it makes fast, cheap judgments. Reranking is the slower, smarter second opinion, and you apply it only to the shortlist.
How rerankers reorder candidates
A bi-encoder, your first stage, embeds the query and each document separately, then compares the vectors. Fast, cheap, and the two texts never actually meet. A cross-encoder reads the query and a candidate document together and attends across both, which produces a far sharper relevance judgment at far higher cost per pair. You cannot run a cross-encoder over a million chunks. You never need to. You run it over the shortlist.
20-100first-stage candidates worth passing to a cross-encoder rerankerTuning top-k at each stage
Retrieve wide, rerank narrow. If your first stage returns only five candidates, the reranker has nothing to work with. You have paid its latency for the privilege of shuffling five items. The classic failure looks like this. The right document sits at position 12 in the first-stage results. The generator only ever sees the top 5. The answer is wrong forever. Widen first-stage top-k to 50, let the reranker lift position 12 to position 2, and the problem vanishes. One caveat, kept light: rerankers trained on web-search data can stumble on specialized corpora, so verify on your own queries before you trust the reorder.
Generation: The Last Mile Where Grounding Breaks
You have retrieved and reranked well. Now the generator has to actually use what you gave it. Grounding breaks here more often than people expect, and it breaks in ways that look nothing alike.
Assembling the context window
Prompt assembly deserves real design. Order chunks so the strongest evidence sits where the model attends best. Attach source labels so the answer can cite inline. And tell the model exactly what to do when the context does not contain the answer. "Answer only from the provided context" is necessary and nowhere near sufficient. Models comply with it about as reliably as toddlers comply with "just one cookie."
When the model ignores your retrieved chunks
Watch for three different breaks, because they need three different fixes. Sometimes the model hallucinates even though the context is good. Sometimes it ignores the context in favor of parametric memory and recites what it learned in training instead of what you retrieved. Sometimes it refuses to answer while the answer sits verbatim in chunk two. Stronger citation requirements help the first. Explicit instructions to prefer context over prior knowledge help the second. Few-shot examples of extraction help the third. Observe them separately. Lumping them into "the bot is wrong sometimes" hides which lever to pull.
How Do You Know Your RAG Pipeline Actually Works?
Vibes-based evaluation, asking it a few questions and nodding, is how quiet failures survive to production. Measure the two halves separately.
Retrieval metrics vs generation metrics
For the retriever, recall@k tells you whether the correct document appears in the top k at all, and MRR tells you how high it ranks. For the generator, faithfulness measures whether the answer sticks to the retrieved context, and answer relevance measures whether it actually addresses the question. Separate them, because the remedies differ completely. Poor recall points at chunking or embeddings. Poor faithfulness points at prompt assembly. Tutorials on datacamp.com walk through implementing these metrics if you want the hands-on version.
Building a golden dataset from real queries
- Before launch, collect 50-100 questions real users would ask, from support tickets, sales calls, and internal Slack threads.
- For each, record the document and passage that contains the correct answer.
- Run every pipeline change against this set and diff the metrics - a chunking tweak that helps one query class routinely hurts another, and only regression testing catches it.
- After launch, grow the set from production traffic, especially the queries that failed.
This evaluation data pays a second dividend. It tells you whether retrieval is your bottleneck or the model itself is, which is the heart of the RAG vs fine-tuning decision. Once your pipeline is measurably solid, the natural next step is agentic RAG, where the system decides for itself when and what to retrieve.
A Worked RAG Pipeline Example: Tracing One Question End to End
Theory is cheap. Trace one realistic query, "what is our refund window for annual plans?", through all six stages, with one failure deliberately injected.
The trace, stage by stage
- Ingest: the refund policy exists twice - the current version and a stale copy from a wiki migration. Deduplication would have caught it here. It did not run.
- Chunk: structure-aware splitting keeps the "Annual Plans" section intact in both copies, each chunk prefixed with "Refund Policy > Annual Plans."
- Embed: both chunks embed almost identically, because the text differs by one number.
- Retrieve: top-50 dense search returns both copies, stale one at position 3, current at position 7. A freshness filter on ingest metadata would have flagged the conflict.
- Rerank: the cross-encoder scores them nearly equally - both are perfectly relevant. Reranking cannot fix a data problem, and a date-aware tiebreaker would have caught what pure relevance cannot.
- Generate: the model cites the stale chunk and confidently reports a 14-day window. The real policy says 30. Fluent, sourced, wrong - and only a golden-set regression test comparing the answer against the known passage flags it.
The lesson lands hard: four separate stages had a chance to catch this, and the fix belongs at the earliest one.
The production readiness checklist
- Spot-check parsed output for 10 documents against their originals
- Confirm deletion and freshness handling actually removes stale content
- Verify permissions metadata is attached at ingest and enforced at retrieval
- Read 20 random chunks and check each stands alone
- Confirm a full reindex followed your last embedding model change
- Test five queries containing exact identifiers like SKUs or error codes
- Measure recall@20 against recall@5 to size your reranking headroom
- Check the model cites sources and the citations resolve to real passages
- Run your golden set and diff against the last release
Put a Grounded Pipeline to Work
Grounded, verifiable answers are why you bother to build this carefully. It is the same principle we build on at AstroFabric. Eight specialist agents compute exact numbers in a code sandbox instead of guessing, gate every write behind your approval, and meet you in the console, over REST or MCP, or right inside Slack. If you care enough about grounding to read this far, start with a free account and see the difference it makes.
Frequently asked questions
What are the main stages of a RAG pipeline?
A production RAG pipeline runs six stages in sequence: ingestion pulls and parses source documents, chunking splits them into retrievable units, embedding converts chunks into vectors, retrieval finds candidates for a query, reranking reorders those candidates with a stronger model, and generation composes the final grounded answer. Each stage has its own failure mode, and problems compound as data moves downstream.
What chunk size should I use for document chunking in RAG?
There is no universal number, but 200-500 tokens with structure-aware boundaries is a sensible starting point for most corpora. Small chunks retrieve precisely but strip context; large chunks preserve context but dilute the embedding. The better move is respecting document structure - never split a table mid-row or a policy mid-sentence - then tuning size against a golden set of real queries.
Do I really need a reranker in my RAG pipeline?
You need one as soon as answer quality matters more than a few hundred milliseconds of latency. First-stage vector search is built for speed across millions of chunks, so the right document often lands at position 8 or 12. A cross-encoder reranker scoring the top 20-100 candidates reliably lifts it into the context window the generator actually sees.
How is a RAG pipeline different from fine-tuning a model?
A RAG pipeline injects fresh knowledge at query time by retrieving documents, while fine-tuning bakes knowledge and behavior into model weights during training. RAG wins when your information changes often, needs citations, or carries access permissions. Fine-tuning wins for style, format, and domain reasoning. Most production teams use RAG first and add fine-tuning only when evaluation data points to it.
How do you evaluate whether a RAG pipeline is working?
Evaluate the two halves separately. Measure retrieval with recall@k and MRR against a golden set of real queries paired with known correct documents. Measure generation with faithfulness (does the answer stick to retrieved context) and answer relevance. Then regression-test both on every change, because a chunking tweak that improves one query class routinely degrades another.
Sources
Every playbook on this blog ships as a runnable mission.
Open a workspace and the playbook library is waiting - describe the outcome and the agents carry it end to end, on your plan's monthly credits.