RAG Evaluation: Separate the Retrieval Failure from the Writing Failure
Your retrieval system gave a wrong answer. There are exactly two reasons, they need opposite fixes, and the one thing that will not tell you which is looking at the answer.
Either the right document was never retrieved, or it was retrieved and the model wrote something unsupported by it. The first is a search problem and prompt engineering will not touch it. The second is a generation problem and re-indexing will not touch it.
The diagnostic is cheap once you know it. Check whether the answer is faithful to the context it received, and separately whether that context contained what was needed. High faithfulness with a bad answer means retrieval failed: the model wrote loyally from the wrong evidence. Low faithfulness means the model ignored evidence it had.
Teams that measure only end-to-end quality tune the wrong half for weeks. The usual pattern is rewriting the prompt repeatedly to fix what is actually a chunking problem, because the prompt is the thing that is easy to change.
Recall is the metric that matters most at the first stage, and this follows from how the system works: a model can ignore irrelevant context, but it cannot write from evidence it never received. Precision is recoverable; recall is not.
Three measurements tell you nearly everything. Did the right chunk make it into the context? Is the answer grounded in what was retrieved? And is the right chunk near the top, rather than merely present?
Measure context recall first, meaning whether the information needed to answer was retrieved at all, because it is the ceiling on everything downstream. Low precision is survivable and low recall is not. Then use order-aware measures, because a correct chunk at position nine has technically been retrieved and practically failed.
Binary retrieval metrics treat a chunk as found or not found. Real systems care where it landed.
Context recall asks whether the information needed to answer was retrieved at all. It is the ceiling on everything downstream and the first thing to measure. If recall is poor, nothing else is worth tuning.
Context precision asks what proportion of what you retrieved was relevant. Low precision costs tokens and dilutes attention, and it is survivable in a way low recall is not.
Order-aware measures are the ones that correlate best with end-to-end quality. Mean reciprocal rank, normalised discounted cumulative gain, precision at k. A system that puts the right chunk at position nine of ten has technically retrieved it and practically failed, because models attend unevenly across a long context.
Position within the context matters, not only rank in the result list. Models attend unevenly across a long window, and evidence placed in the middle of a large context is used less reliably than the same evidence at either end. If you pass many chunks, where you put the best one is a decision worth testing rather than leaving to the retriever's sort order.
Which means reranking is often the highest-return change available. Retrieve generously to protect recall, then rerank to put the best evidence first. This decouples the two metrics rather than trading them, and it is a smaller change than re-architecting the index.
Hybrid retrieval is the default worth starting from. Dense vectors handle paraphrase and meaning; keyword search handles exact terms, product codes, error numbers and names, which embeddings routinely miss. Running both and merging covers each one's blind spot, and teams that start with vectors alone usually end up adding keyword search after a user searches for a serial number and gets nothing.
Measure at the k you actually use. Retrieval quality at ten documents tells you little if you pass three to the model.
Faithfulness checks whether the generated answer is supported by the retrieved context. It is the single most useful generation metric and it is routinely misread.
It measures grounding, not correctness. An answer perfectly faithful to a retrieved document that is out of date, wrong, or about a different product scores well and is wrong. Faithfulness tells you the model behaved; it does not tell you the system was right.
Which is exactly why it is diagnostic. High faithfulness plus a wrong answer isolates the problem to retrieval or to the corpus. That inference is worth more than the score itself.
Answer relevance is the complementary check: did the response address the question that was asked, rather than an adjacent one it had better evidence for. Models are prone to answering the question they can answer.
The hardest case for any of these is the correct refusal. When the corpus genuinely does not contain the answer, the right behaviour is to say so. Most evaluation sets contain no such cases, so systems are never scored on the one behaviour that prevents confident fabrication. Add them deliberately.
Thresholds published in vendor guides are starting points, not standards. Figures like a faithfulness of 0.75 or precision at five of 0.7 circulate widely and are not standardised benchmarks. Set your own by sampling what each score means on your content.
Chunk size and overlap are usually chosen once, early, by whoever set up the pipeline, and then never revisited despite being among the highest-impact settings.
Test rather than reason about it. Published practice suggests trying 256, 512 and 1024 tokens with varying overlap, re-indexing for each, and measuring. The right answer depends on your documents and cannot be derived from first principles.
Each configuration requires a full re-index, which is why teams avoid testing it and why the test is worth automating early. A pipeline that can re-index on demand turns a week of argument into an afternoon of measurement.
The failure mode of small chunks is lost context. A paragraph that makes sense only after the heading above it retrieves alone and means nothing. Parent-child retrieval addresses this: match on the small chunk, return the larger passage around it.
The failure mode of large chunks is dilution. One relevant sentence arrives surrounded by a page of irrelevance, which costs tokens and buries the signal.
Structure beats size where documents have structure. Splitting on headings, sections or logical units usually outperforms a fixed token count, because it respects boundaries the author already decided were meaningful.
Most retrieval evaluation assumes a well-formed question. Users do not supply one, and the gap between the query typed and the query that would retrieve well is a fixable loss.
Short queries carry too little signal. Two words match many things weakly. Expanding them with context from the conversation or the user's situation frequently improves recall more than any index change.
Follow-up questions are incomplete by design. "What about the second one?" contains no retrievable content at all. Without rewriting against the conversation history, the retriever searches for nothing useful and the system looks stupid at exactly the moment the user was engaged.
Multi-part questions retrieve badly as one query. A question containing two asks pulls documents that are mediocre for both rather than good for either. Splitting, retrieving separately and combining is usually better.
Jargon mismatch is the quiet one. Users ask in their words and documents are written in yours. A glossary mapping the two, applied at query time, is cheap and often more effective than a better embedding model.
Evaluate the rewrite separately. It is a component with its own failure mode, and a rewrite that drops a constraint from the original question produces a confidently retrieved wrong answer.
If different users may see different documents, the retrieval layer has become an access control surface, and the naive implementations are wrong in ways that are hard to notice.
Filtering after retrieval is the wrong order. Retrieving ten and discarding the six the user cannot see leaves four, which may not include what they needed. The result is a system that works well for administrators and badly for everyone else, which is exactly the pattern internal testing fails to catch.
Filter inside the search, not after it. The index has to support per-document permission metadata so the user's own ten are retrieved. This is a requirement to set when choosing infrastructure, not a layer to add later.
Permissions change and indexes go stale. Someone leaves a team, a document is restricted, a project closes. If the index holds a copy of the permission state, it must be refreshed, and the window between a change and a re-index is a window of exposure.
The answer itself can leak. Even with correct filtering, a summary synthesised across documents can reveal something no single permitted document stated. This is rare, it is real, and it is the argument for not using one shared index across very different trust levels.
Test with a low-privilege account. Most retrieval systems are built and demonstrated by people with broad access, so the restricted experience is never seen until a user reports it.
Everyone measures the retriever and the model. Almost nobody measures the thing being searched, which is frequently where the real defect lives.
If the answer is not in the corpus, no amount of tuning finds it. A surprising share of reported retrieval failures are coverage failures, and the only way to know is to check, by hand, whether the document that should answer the question exists at all.
Stale content is worse than missing content. A missing document produces a refusal or a vague answer. An outdated document produces a confident, well-grounded, wrong one, and faithfulness scores it highly.
Contradictions are invisible until they surface. Two documents saying different things about the same policy means retrieval picks one, essentially at random, and the system is right half the time with no indication which half.
Duplicates consume the context window. Near-identical documents crowd out diversity, so the model receives the same fact five times instead of five facts.
Metadata is retrieval signal that most corpora waste. Document date, owner, product, status and version are usually available and usually unused. Filtering on them before the semantic search narrows the field honestly, and surfacing them in the context lets the model say when something was last updated rather than stating stale information as current.
Per-chunk analytics are the cheapest available improvement. Track how often each chunk is retrieved and how often it contributes to a good answer. Chunks retrieved frequently and never helpful are mis-chunked or misleading, and that list prioritises your re-chunking work for you.
The set determines everything, and the common mistake is writing it rather than collecting it.
Collect real questions. What people actually type, including the vague, the misspelled and the ones containing two questions at once. Invented questions are cleaner than reality in exactly the ways that matter.
Label the supporting evidence, not just the answer. For each question, which document or chunk should have been retrieved. Without that you can only measure end-to-end and lose the diagnostic.
Include unanswerable questions, as above, with the expected behaviour being an explicit refusal.
Date the set. Questions whose correct answer changes over time need a recorded as-of date, or the set starts failing for reasons that have nothing to do with the system.
Include near-misses deliberately: questions where a plausible wrong document exists in the corpus. These are where reranking earns its place and where naive retrieval fails quietly.
Have two people label a sample independently. Where they disagree about which document should answer a question, the question is ambiguous and so is your system's task. That disagreement is information about the product, not noise in the dataset.
Keep it small and current. A set of fifty well-labelled questions, maintained, beats a thousand written once and never revisited. The cost of a stale evaluation set is that it blocks correct changes, which teaches everyone to ignore it.
Offline evaluation catches regressions. Production tells you what you got wrong when you built the set.
Sample production traffic continuously. Published practice suggests reviewing a small percentage of live traffic on an ongoing basis while running the full set in continuous integration. The exact proportion matters less than that somebody looks.
The production signals worth instrumenting need no judge and come from logs: refusal rate, retrieval latency, number of chunks retrieved, and whether users rephrased immediately, which is the clearest available signal of a bad answer.
Watch the refusal rate in both directions. Rising means coverage is degrading or the threshold is too strict. Falling sharply often means the system has started answering things it should not.
Feed every production failure into the offline set, which is how the two compound rather than existing separately.
Alert on distribution shift in the questions themselves. A sudden change in what users are asking usually means something changed in the product or the world, and your corpus has not caught up yet.
Re-run everything when the embedding model changes. A new embedding model means a full re-index and invalidates your tuning, and it is a bigger change than a model upgrade on the generation side.
A retrieval system that is accurate and slow is a product problem, and the trade-offs are adjustable.
Retrieval adds latency before generation starts. Embedding the query, searching, reranking and assembling context all happen before the first token, and that delay is visible to the user.
Reranking improves quality and costs time. A cross-encoder reranker is substantially better and substantially slower than retrieving alone, and whether it is worth it is a measurement rather than a principle.
More context is not free. Passing ten chunks instead of three triples the input tokens on every call, and the quality gain usually flattens well before the cost does. Measure where your curve flattens and set k there.
The embedding step has its own cost at scale. Indexing a large corpus is a one-off bill, re-indexing on a chunking change repeats it, and embedding every incoming query is a per-request cost that is small individually and visible at volume.
Caching helps more here than people expect, since a stable system prompt and a stable retrieved prefix are exactly what prompt caching is for.
Assert latency and cost in the evaluation, not just quality, or the pipeline will drift toward expensive and slow one well-intentioned change at a time.
Context recall on fifty real labelled questions. If the right chunk is not arriving, nothing else matters and everything else is premature.
Faithfulness on the same set, to split the remaining failures into retrieval and generation.
Rank position of the correct chunk, which usually reveals that reranking is the cheapest available win.
Refusal behaviour on unanswerable questions, because confident fabrication is the failure that damages trust fastest.
Then chunking as a measured experiment, three sizes, re-indexed, scored.
Our rate band is published: $40 to $100 an hour by role, blending to $60 to $70 for a mixed team. The thing we would say before any of it is that most teams asking us to improve their RAG prompt have a retrieval problem, and the measurement that distinguishes them takes a day.
In two layers, because a wrong answer has two possible causes needing opposite fixes. Measure retrieval separately: did the information needed to answer get into the context at all, and how highly was it ranked. Then measure generation: is the answer grounded in what was retrieved. End-to-end scores alone cannot distinguish a search failure from a writing failure, which is why teams rewrite prompts for weeks to fix chunking problems.
Whether the generated answer is supported by the retrieved context. Crucially it measures grounding rather than correctness: an answer faithful to a document that is stale or about a different product will score highly and still be wrong. Its real value is diagnostic — high faithfulness alongside a wrong answer points the problem at retrieval or at the corpus rather than at the model, which is exactly the split you need.
Recall, at the first stage. A model can ignore irrelevant context, but it cannot write from evidence it never received, so a missed chunk is unrecoverable while a noisy one is merely expensive. The practical pattern is to retrieve generously to protect recall and then rerank to restore precision and ordering, which decouples the two rather than trading one against the other.
Test rather than decide. Published practice suggests trying 256, 512 and 1024 tokens with varying overlap, re-indexing for each and measuring. Small chunks lose the context that made a passage meaningful, which parent-child retrieval addresses by matching on the small chunk and returning the surrounding passage. Large chunks dilute signal and cost tokens. Where documents have real structure, splitting on headings usually beats any fixed token count.
Usually because it retrieved a document that is stale, contradicted elsewhere in the corpus, or about something adjacent, and then answered faithfully from it. Faithfulness scores that behaviour highly, which is why corpus quality needs evaluating as its own component. The second cause is that the evaluation set contains no unanswerable questions, so the system has never been scored on refusing, which is the behaviour that prevents fabrication.
Smaller than people expect, if it is well built. Around fifty real questions, collected from actual usage rather than invented, with the supporting document labelled for each so retrieval can be scored separately. Include unanswerable questions with refusal as the expected behaviour, and near-misses where a plausible wrong document exists. A maintained set of fifty beats a thousand written once, because a stale set blocks correct changes and teaches everyone to ignore it.
Signals available from logs without a judge: refusal rate, retrieval latency, number of chunks retrieved, and whether users immediately rephrase, which is the clearest free indicator of a bad answer. Watch refusal rate in both directions, since rising means coverage is degrading and a sharp fall often means the system has started answering things it should not. Sample a proportion of live traffic for human review and feed every failure back into the offline set.
Vikas has around fifteen years of experience building software and now builds generative AI systems at Zyneto. His work covers retrieval augmented generation, agentic AI, knowledge graphs, AI memory, and the evaluation and guardrails that decide whether any of it is safe to put in front of customers. He has shipped enterprise copilots, document AI, chatbots and predictive analytics for e-commerce, fintech and marketing teams, and works day to day in Python, JavaScript and SQL. He follows multimodal models, business process automation and enterprise AI security closely, and mentors engineers moving into AI. He writes about architecture, inference cost and the failure modes that only show up at production scale.
Share your details and we will talk soon.
Be the first to access expert strategies, actionable tips, and the trends actually shaping the digital world. No fluff - just practical insights delivered straight to your inbox.
Dive into our blog and stay ahead of the curve with expert perspectives, future-ready trends, and tech tips written for decision-makers and doers alike.
