Building RAG Pipelines That Survive Production

Retrieval-augmented generation demos beautifully and fails quietly. The gap between a notebook that answers questions about ten PDFs and a system that serves an entire organisation is not model quality — it is retrieval quality, evaluation discipline, and operational plumbing.

This is the architecture I keep arriving at after building these systems for engineering and support teams, and the specific decisions that matter.

Why naive RAG breaks#

The canonical pipeline is four steps: chunk the documents, embed the chunks, retrieve the top k by cosine similarity, stuff them into the prompt. It works on a small, homogeneous corpus. It degrades badly once you add real documents.

Three failure modes dominate:

Chunk boundaries destroy meaning. A fixed 512-token window splits a table from its header, a configuration block from the paragraph explaining it, a warning from the procedure it applies to. The retrieved chunk is topically relevant and practically useless.

Semantic similarity is not relevance. Embeddings capture topic, not intent. Ask "why did the upgrade fail?" and you retrieve every document that discusses upgrades. The one incident report that actually contains the answer ranks eleventh.

No grounding signal. The model receives five chunks with no indication of which is authoritative, which is current, and which is a three-year-old forum post someone pasted into the wiki.

The most common production incident I see is not hallucination — it is confident retrieval of a superseded document. A decommissioned runbook that still lives in the index will be cited happily and forever.

The architecture that holds up#

Each stage exists because removing it caused a measurable regression. Let me take the ones that matter most.

Structure-aware chunking#

Stop chunking by token count. Chunk by document structure, then split oversized sections with overlap.

For Markdown and HTML, split on heading boundaries and carry the heading path into the chunk:

def chunk_by_structure(doc: Document, max_tokens: int = 900) -> list[Chunk]:
    chunks = []
    for section in split_on_headings(doc):
        # The heading path gives every chunk standalone context.
        prefix = " > ".join(section.heading_path)
        body = section.text

        if count_tokens(body) <= max_tokens:
            chunks.append(Chunk(text=f"{prefix}\n\n{body}", meta=section.meta))
            continue

        # Oversized sections split on paragraph boundaries with overlap,
        # never mid-sentence.
        for window in sliding_paragraphs(body, max_tokens, overlap=120):
            chunks.append(Chunk(text=f"{prefix}\n\n{window}", meta=section.meta))

    return chunks

That heading prefix is the single highest-leverage change in this entire pipeline. A chunk reading Deployment > Rollback > Database migrations\n\nRun the down migration before… is retrievable and interpretable. The same text without its path is neither.

Tables get special handling: never split a table, and serialise it with its caption and column headers repeated. If a table exceeds the window, emit one chunk per row group with the header attached.

Hybrid retrieval, because embeddings miss exact terms#

Vector search cannot reliably find ORA-01555, KB5034441, or vmnic3. Lexical search finds them immediately and has no idea that "the box won't boot" relates to "host fails POST".

Run both and fuse the rankings. Reciprocal rank fusion is the workhorse — no score calibration needed, which matters because BM25 scores and cosine distances are not comparable:

def reciprocal_rank_fusion(rankings: list[list[str]], k: int = 60) -> list[str]:
    scores: dict[str, float] = defaultdict(float)
    for ranking in rankings:
        for position, doc_id in enumerate(ranking):
            scores[doc_id] += 1.0 / (k + position + 1)
    return sorted(scores, key=scores.get, reverse=True)

Retrieve 50 candidates from each index, fuse, then pass the top 25 to a reranker.

Reranking is where accuracy is won#

A cross-encoder scores the query and document together rather than comparing independent embeddings. It is far slower per pair and far more accurate, which is exactly the right trade for a final-stage filter over 25 candidates.

In every system I have measured, adding a reranker moved answer accuracy more than upgrading the generation model. It is also the cheaper change.

Those numbers are from an internal knowledge base of roughly 40,000 documents. Your corpus will differ; the ordering rarely does.

Query rewriting#

Users type fragments. "teams call quality bad" needs to become something a retriever can work with. A small, fast model handles this well:

Rewrite the user's question into a standalone search query.
Expand abbreviations and add relevant technical synonyms.
Output only the rewritten query.

User: teams call quality bad
Rewritten: Microsoft Teams poor call quality jitter packet loss audio degradation troubleshooting

For conversational interfaces this step is mandatory rather than optional — follow-up questions like "what about on wireless?" are meaningless without the preceding turn folded in.

Metadata and freshness#

Every chunk carries structured metadata, and the retriever filters on it before ranking:

{
  "source_system": "confluence",
  "space": "infrastructure",
  "url": "https://wiki.internal/x/AB12",
  "last_modified": "2026-04-11T09:22:00Z",
  "review_status": "current",
  "owning_team": "platform-networking",
  "classification": "internal"
}

Two of these do disproportionate work.

review_status lets you exclude documents an owner has marked superseded — the single most effective fix for the stale-runbook problem above.

classification enforces access control at retrieval time. Filtering after generation is far too late; the content is already in the context window and, depending on your logging, already on disk.

Access control belongs in the retrieval filter, evaluated against the calling user's permissions on every request. A shared index with post-hoc filtering is a data leak waiting for its first curious employee.

Evaluation: the part everyone skips#

You cannot improve what you do not measure, and "it seems better" is not measurement. Build a golden set of 150–300 real questions with verified answers and the document IDs that contain them. Real questions, from real logs — not questions you invented, which are always easier than what users actually ask.

Then track three things separately:

Retrieval recall@k — did the correct document appear in the retrieved set? This isolates the retriever from the generator. If recall@5 is 0.6, no amount of prompt engineering will save you.

Answer correctness — graded against the reference answer. An LLM judge with a strict rubric correlates well enough with human grading to be useful for regression testing, provided you validate the judge against human labels once.

Citation validity — does every claim in the answer trace to a retrieved chunk? This catches the failure mode users find most damaging: a fluent answer citing a real document that does not actually support the statement.

Run the suite in CI on every prompt, model, or chunking change. A 40-minute evaluation run that blocks a regression is cheap.

Cost control#

Two levers matter and both are unglamorous.

Cache aggressively. Embedding the same document twice is pure waste; hash the normalised content and skip unchanged chunks on re-ingestion. Prompt caching on the generation side pays for itself immediately when your system prompt and retrieved context share a stable prefix.

Route by difficulty. Most queries are straightforward lookups a small model answers perfectly. Reserve the frontier model for queries that fail a confidence check or explicitly request analysis. Classify with a cheap model, route accordingly, and measure the accuracy delta so the routing threshold is a data-driven decision rather than a guess.

What I would build first#

If you are starting today and want the shortest path to something genuinely useful:

1. Structure-aware chunking with heading paths. Half a day, largest single accuracy gain.
2. Hybrid retrieval with RRF. One day, closes the exact-match gap.
3. A golden evaluation set of 150 real questions. Two days, and it makes every subsequent decision measurable.
4. Cross-encoder reranking. Half a day, biggest accuracy gain per unit of effort after chunking.
5. Metadata filtering with review_status and access classification. One day, and it is what makes the system safe to expose beyond a pilot group.

Everything after that — agentic multi-hop retrieval, query decomposition, graph-augmented context — is genuinely valuable, and genuinely premature until these five are solid.