How to Build a RAG Pipeline: From Ingestion to Answer

A RAG pipeline is the production system that turns raw documents into grounded AI answers through six stages: ingestion, chunking, embedding, retrieval, reranking, and generation. Every stage of the rag pipeline has a distinct failure mode, and errors compound downstream, so a fluent final answer can hide a broken step three stages back. This walkthrough shows what each stage does, where it breaks in practice, and how to trace a real query end to end so you can find the weak link before your users do.

Want the conceptual grounding first? Our explainer on Retrieval-Augmented Generation covers what RAG is and why it exists. This post is the part that piece deliberately leaves out: the operational machinery, stage by stage, and the specific ways each stage betrays you in production.

Think of the pipeline as a relay race. Every runner trusts the baton they were handed. Nobody checks it mid-race.

Here is the trap. A broken calculator returns garbage you notice immediately. A broken RAG pipeline returns beautiful prose. I have watched a support bot quote a deprecated pricing page with perfect confidence, the formatting intact, a friendly line to close. Ingestion had added documents and never expired them, so the old page embedded just as convincingly as the new one. The user experience was flawless right up until the customer asked for the discount the bot promised.

Everything downstream inherits whatever ingestion produces. This is the highest-leverage stage, and still the one that gets skipped.

Real corpora are ugly. PDFs mangle tables into single-column soup. HTML drags navigation bars and cookie banners into the extract. OCR turns "10%" into "1O%". The same policy document often lives in three slightly different versions across a wiki, a shared drive, and someone's export folder. Those near-duplicates are the nasty ones. They crowd retrieval with redundant chunks and shove genuinely different documents out of the top-k.

Capture metadata at ingest time, because retrofitting it means re-crawling everything. At minimum:

• Source URL or file path, so answers can cite something clickable
• Publication and last-modified dates, so freshness can be scored
• Access permissions, so a sales rep never retrieves an HR document
• Document type and section hierarchy, for structure-aware chunking later

Build deletion in from day one. A pipeline that only adds documents drifts toward stale answers within months, quietly, one outdated policy at a time.

Chunking looks like a boring preprocessing detail. It is the decision that most shapes retrieval quality, because chunks are the atomic unit everything else operates on.

The naive approach slices text every 500 tokens, indifferent to what the text is doing. Predictable wreckage follows. A pricing table splits mid-row, and the chunk that says "Enterprise tier" no longer sits with the chunk that holds its price. Retrieval finds one half. The generator improvises the other. You get an answer that references half a policy with total confidence. Structure-aware splitting keeps headings with their sections, honors list boundaries, and refuses to shred a table. It is worth every hour it takes to build.

Small chunks embed precisely but strip context. Large chunks preserve context but dilute the embedding until nothing matches strongly. Most teams land somewhere between 200 and 500 tokens, with modest overlap so a sentence that straddles a boundary still survives. Two tricks punch above their weight. Prepend each chunk with its document title and section breadcrumb. And look at parent-child chunking, where you retrieve on the small chunks but hand the generator the larger parent section they came from.

An embedding model compresses a chunk into a vector, and semantic similarity becomes geometric proximity. That is the whole trick. It works remarkably well inside the domain the model was trained on.

Feed dense legal text or clinical notes through a general-purpose model and recall degrades quietly. The model never learned that two pieces of jargon mean the same thing. You do not need weeks of benchmarking theater to catch this. Take fifty real queries, embed them with two or three candidate models, and count how often the known correct document lands in the top ten. Resources like IBM's material on retrieval architectures are useful for the landscape. Your own fifty queries still beat any public leaderboard. One operational rule is non-negotiable. If you ever swap embedding models, reindex everything. Vectors from different models live in incompatible spaces, and mixing them corrupts every similarity score.

Dense vectors are brilliant at meaning and terrible at exact identifiers. Ask for "error E4013" and semantic search happily returns documents about errors in general. Keyword search nails the identifier and then misses every paraphrase. Hybrid retrieval fuses dense and sparse scores so you get both. For a corpus full of SKUs, error codes, or part numbers, that fusion is the difference between a useful system and a frustrating one. On the index itself, stay with exact search until scale genuinely forces approximate nearest neighbors. When it does, tune ANN settings with recall measurements in hand rather than defaults on faith.

First-stage retrieval is built for speed across millions of chunks, so it makes fast, cheap judgments. Reranking is the slower, smarter second opinion, and you apply it only to the shortlist.

A bi-encoder, your first stage, embeds the query and each document separately, then compares the vectors. Fast, cheap, and the two texts never actually meet. A cross-encoder reads the query and a candidate document together and attends across both, which produces a far sharper relevance judgment at far higher cost per pair. You cannot run a cross-encoder over a million chunks. You never need to. You run it over the shortlist.

Retrieve wide, rerank narrow. If your first stage returns only five candidates, the reranker has nothing to work with. You have paid its latency for the privilege of shuffling five items. The classic failure looks like this. The right document sits at position 12 in the first-stage results. The generator only ever sees the top 5. The answer is wrong forever. Widen first-stage top-k to 50, let the reranker lift position 12 to position 2, and the problem vanishes. One caveat, kept light: rerankers trained on web-search data can stumble on specialized corpora, so verify on your own queries before you trust the reorder.

You have retrieved and reranked well. Now the generator has to actually use what you gave it. Grounding breaks here more often than people expect, and it breaks in ways that look nothing alike.

Prompt assembly deserves real design. Order chunks so the strongest evidence sits where the model attends best. Attach source labels so the answer can cite inline. And tell the model exactly what to do when the context does not contain the answer. "Answer only from the provided context" is necessary and nowhere near sufficient. Models comply with it about as reliably as toddlers comply with "just one cookie."

Watch for three different breaks, because they need three different fixes. Sometimes the model hallucinates even though the context is good. Sometimes it ignores the context in favor of parametric memory and recites what it learned in training instead of what you retrieved. Sometimes it refuses to answer while the answer sits verbatim in chunk two. Stronger citation requirements help the first. Explicit instructions to prefer context over prior knowledge help the second. Few-shot examples of extraction help the third. Observe them separately. Lumping them into "the bot is wrong sometimes" hides which lever to pull.

Vibes-based evaluation, asking it a few questions and nodding, is how quiet failures survive to production. Measure the two halves separately.

For the retriever, recall@k tells you whether the correct document appears in the top k at all, and MRR tells you how high it ranks. For the generator, faithfulness measures whether the answer sticks to the retrieved context, and answer relevance measures whether it actually addresses the question. Separate them, because the remedies differ completely. Poor recall points at chunking or embeddings. Poor faithfulness points at prompt assembly. Tutorials on datacamp.com walk through implementing these metrics if you want the hands-on version.

1. Before launch, collect 50-100 questions real users would ask, from support tickets, sales calls, and internal Slack threads.
2. For each, record the document and passage that contains the correct answer.
3. Run every pipeline change against this set and diff the metrics - a chunking tweak that helps one query class routinely hurts another, and only regression testing catches it.
4. After launch, grow the set from production traffic, especially the queries that failed.

This evaluation data pays a second dividend. It tells you whether retrieval is your bottleneck or the model itself is, which is the heart of the RAG vs fine-tuning decision. Once your pipeline is measurably solid, the natural next step is agentic RAG, where the system decides for itself when and what to retrieve.

Theory is cheap. Trace one realistic query, "what is our refund window for annual plans?", through all six stages, with one failure deliberately injected.

1. Ingest: the refund policy exists twice - the current version and a stale copy from a wiki migration. Deduplication would have caught it here. It did not run.
2. Chunk: structure-aware splitting keeps the "Annual Plans" section intact in both copies, each chunk prefixed with "Refund Policy > Annual Plans."
3. Embed: both chunks embed almost identically, because the text differs by one number.
4. Retrieve: top-50 dense search returns both copies, stale one at position 3, current at position 7. A freshness filter on ingest metadata would have flagged the conflict.
5. Rerank: the cross-encoder scores them nearly equally - both are perfectly relevant. Reranking cannot fix a data problem, and a date-aware tiebreaker would have caught what pure relevance cannot.
6. Generate: the model cites the stale chunk and confidently reports a 14-day window. The real policy says 30. Fluent, sourced, wrong - and only a golden-set regression test comparing the answer against the known passage flags it.

The lesson lands hard: four separate stages had a chance to catch this, and the fix belongs at the earliest one.

Put a Grounded Pipeline to Work

Applying this to business intelligence

AstroFabric is useful when your desired output is verified company and person data. Its agents research and enrich records and deliver structured results into existing systems. If your project instead needs a custom document ingestion and retrieval service, evaluate and operate that infrastructure separately.

Put the workflow to a small test

Choose one objective and a small sample. Set a credit ceiling, inspect the evidence and missing fields, then review the proposed destination write. Start with AstroFabric, or read the API and MCP documentation. See current plans and credit pricing before increasing volume.

Frequently asked questions

Sources

• ibm.com
• datacamp.com