Why RAG systems fail in production, and the five fixes that matter

We get called into RAG rescues fairly often. The pattern is consistent enough to be predictable: a team built something that demoed well, put it in front of real users, and watched the answer quality collapse. By the time we arrive they have usually tried three models, rewritten the system prompt four times and added a paragraph telling the model not to make things up.

None of that helps, because in almost every case the problem is upstream of the model. If the passage containing the answer never reaches the context window, no prompt can recover it. The model is doing exactly what it was asked to do with what it was given.

Measure retrieval before you touch generation

The single most useful thing you can do to a struggling RAG system is stop evaluating the final answer for a week and evaluate retrieval instead.

Take fifty real questions from your users. For each one, have a subject expert identify which document, and ideally which passage, contains the answer. Then ask one question of your system: is the correct passage in the top k results?

That number is your recall ceiling. If retrieval recall is 60%, then 40% of your answers are guaranteed to be wrong or hedged no matter what model you put on top, and every hour spent on prompt engineering is an hour spent on the wrong problem. We have seen systems where the reported answer accuracy was 55% and retrieval recall was 58%. The model was contributing almost nothing to the failure.

1. Chunking that destroys the meaning

Fixed-size splitting is the default in every tutorial and it is the first thing to go. A 512-token window applied blindly cuts a table away from its header, separates a contract clause from the definition it depends on, and splits a numbered procedure across two chunks so that neither one is answerable on its own.

What to do instead depends on your documents, which is why this cannot be decided from a blog post:

• Structured documents such as contracts, policies and manuals chunk best on their own structure. Section and clause boundaries carry meaning that character counts do not.
• Tables need to travel with their headers and usually with their caption. A row of numbers with no column names is noise in an embedding space.
• Scanned material needs layout-aware parsing before chunking is even a question. Multi-column PDFs read in the wrong order produce chunks that are grammatically fluent and semantically scrambled.
• Short records such as tickets and CRM notes often should not be chunked at all. Splitting a 200-word ticket in half rarely helps anyone.

Whatever you choose, test it against your benchmark rather than assuming. Chunking strategy is the variable with the largest effect on recall in most of the systems we have measured, and the correct answer varies by corpus.

2. Single-vector search on its own

Dense embeddings are good at meaning and bad at exactness. Ask a legal system about a specific statute reference, or a support system about a part number, and semantic similarity will confidently return five passages about the general topic and miss the one containing the literal string.

Keyword search has the opposite profile. It finds the exact token and misses the paraphrase.

Hybrid retrieval, running BM25 alongside dense vector search and merging the results, is not an optimisation. For any corpus containing identifiers, product codes, statute references, names or version numbers, it is the difference between a system people trust and one they stop using. Add metadata filtering on top, so a query scoped to one jurisdiction or one department narrows the candidate set before scoring rather than after.

3. No reranking stage

Retrieval returns candidates in embedding-distance order. That ordering is a rough proxy for relevance and it is often wrong in the top few positions, which is exactly where it matters. The passage that answers the question sits at rank seven, and you passed the top five to the model.

A cross-encoder reranker scores each candidate against the actual query rather than comparing two independent embeddings. It is slower, because it runs a model over every pair, which is why you retrieve broadly and rerank narrowly: pull 50 candidates, rerank them, pass the best 5.

In our measurements this is consistently where the largest single precision gain comes from. On one enterprise knowledge platform, adding reranking to an existing hybrid setup lifted retrieval precision 61% over the prior tooling. measured · knowledge-search · prod

4. No ground-truth benchmark, so nobody can tell whether changes help

This is the failure that makes the other four permanent. Without a labelled question set, every change is a matter of opinion. Someone tries a different embedding model, a few spot checks look better, it ships. Two weeks later quality is worse and nobody can say which change did it.

A benchmark does not need to be large to be useful. Fifty to a hundred questions with expert-verified answers, covering your real query distribution including the awkward ones, is enough to make decisions with. Build it in week one, before any architecture is settled, because it is the instrument you will use to settle the architecture.

Then score against it on every change. Faithfulness, answer relevance, context precision and context recall are the four that matter most; frameworks such as RAGAS will compute them for you. Publish the numbers where your team can see them.

5. Access control added after the system works

This one does not degrade answer quality. It kills the project.

A pilot goes well, leadership asks to roll it out company-wide, and someone from legal asks whether a junior analyst can retrieve a board paper. If permissions were handled by asking the model nicely in the system prompt, the honest answer is yes, and the rollout stops.

Access control belongs at retrieval time. Permissions filter the candidate set before scoring, so a document the user cannot read is never a candidate and never reaches the context window. This is straightforward if it is in the design from the start and a substantial rebuild if it is not, because it changes how you index, how you store metadata and how you cache.

The same applies to audit trails. In regulated environments you need a record of every query, the passages returned and the answer generated. Retrofitting that after launch means losing the history you most want to inspect.

What to fix first

If you are looking at a system that underperforms, work in this order:

1. Build the benchmark. One or two days with a subject expert. Without it you are guessing.
2. Measure retrieval recall. This tells you immediately whether you have a retrieval problem or a generation problem, and it is usually the former.
3. Add hybrid search if you are running dense-only and your corpus contains any exact identifiers.
4. Add reranking. Cheapest large precision gain available.
5. Revisit chunking against the benchmark. Slower work, but often the biggest ceiling lift.
6. Then look at the model, the prompt and the generation settings.

Most teams do this list backwards, starting at step six because it is the most visible and the easiest to change. That is why so many RAG projects spend months getting nowhere.