RAG Evaluation: Diagnose Retrieval and Generation Failures Separately

A support assistant gives the wrong refund policy. The model team blames hallucination, the search team blames chunking, and the vendor recommends a larger model. All three can sound plausible because the team has one dashboard number called "RAG accuracy."

That number is hiding the failure.

A retrieval-augmented generation system has at least two distinct jobs. It must find the evidence, then produce an answer grounded in that evidence. A correct-looking answer can come from weak retrieval and model memory. A wrong answer can come from perfect retrieval that the model ignored. Treating both as one score sends engineers toward expensive fixes that do not touch the broken component.

Evaluate RAG as a pipeline. Measure retrieval, generation, and the business result separately. Then every failed test points toward an owner and a repair.

"RAG Accuracy Is Bad" Is Not a Diagnosis

Suppose an employee asks, "Can a director approve a $12,000 customer credit?" The authoritative policy says directors may approve up to $10,000. The assistant says yes.

At least six different defects could produce that answer:

1. The current policy was never indexed.
2. A permission filter excluded it by mistake.
3. Search retrieved an older policy with a $15,000 limit.
4. Search found both versions but ranked the stale one first.
5. The correct passage reached the model, but the model relied on the distracting passage.
6. The model used the right passage and still generated an unsupported conclusion.

These defects need different work. Re-indexing fixes the first. Access-control tests fix the second. Freshness and source ranking fix the third and fourth. Prompting, context layout, or a different generator may fix the fifth and sixth.

RAGChecker, a NeurIPS 2024 evaluation framework, makes this separation explicit. The RAGChecker paper evaluates retrievers with claim recall and context precision, then evaluates generators with measures such as context use, faithfulness, hallucination, and sensitivity to noise. That vocabulary is useful because it maps directly to a production pipeline.

The paper also found a tradeoff that simple benchmarks miss. Increasing retrieved chunks from 5 to 20 raised claim recall from 61.5 to 77.6 and faithfulness from 88.1 to 92.2, but noise sensitivity rose from 34.0 to 35.4. End-to-end F1 moved only from 51.7 to 53.4. A 16.1-point retrieval gain produced a 1.7-point answer gain because more context supplied both evidence and irrelevant material.

That is why "retrieve more" is not a complete strategy. Retrieval needs enough recall to include the answer and enough precision to keep the evidence set usable.

The same principle applies before generation. The BEIR benchmark showed that retrieval methods behave differently across domains and tasks. A configuration that performs well on a public benchmark can still miss your product names, policy language, acronyms, and document structure. Your own questions and sources must decide what works.

The first rule of RAG evaluation is simple: attach every score to a stage.

Build a Claim-Level Evaluation Set

Most teams begin with a spreadsheet containing questions and ideal prose answers. That is better than testing by intuition, but it still makes diagnosis difficult. If an ideal answer has five factual claims and the model gets four right, a single pass or fail label throws away useful information.

Build the evaluation set around claims and sources instead.

For the credit-limit example, the record might contain:

Start with 100 to 200 questions drawn from actual search logs, tickets, employee requests, and subject-matter expert interviews. Do not fill the set with tidy FAQ questions. A useful mix includes:

• Frequent questions that account for most usage
• High-cost errors involving money, contracts, privacy, or customer commitments
• Multi-claim questions requiring evidence from more than one source
• Stale and conflicting documents
• Queries with no supported answer
• Similar documents belonging to different products, countries, or customers
• Permission-boundary cases run as several user roles
• Abbreviations, misspellings, and internal terminology

Each expected claim should point to an authoritative passage. That lets an evaluator ask two separate questions: did retrieval return the passage, and did the response express the supported claim? It also prevents a model from receiving credit for a plausible answer drawn from pretraining rather than company evidence.

Source labels matter as much as answer labels. If the expected answer is correct but cites a stale wiki page instead of the governed policy, the system has not passed. It may fail as soon as those sources diverge. Our guide to retrieval paths and source priority explains how to define which system wins when sources conflict.

Keep the test set out of prompt examples and tuning data. Segment it by workflow, source type, risk, and difficulty so an improved average cannot hide a severe regression. A ten-point gain on easy questions does not compensate for a new leak across customer tenants.

The evaluation set is not a one-time deliverable. Every confirmed production failure should become a regression case. Every reviewer correction is a candidate label. This turns daily operations into a growing record of what the system must continue to do.

Measure Retrieval Before You Judge the Answer

Run retrieval by itself. Give the retriever each question, save the ranked chunks with metadata, and score the evidence before any generator writes an answer.

Four metrics cover most enterprise diagnosis:

Claim recall is the right starting point. If a question requires three claims and returned context supports two, recall is two-thirds. Document recall can overstate performance because one "correct" document may not contain the required section, and one answer may require several documents.

Context precision then asks how much junk came along for the ride. Duplicate chunks, navigation text, stale versions, unrelated products, and near matches consume the model's attention. Research on the lost-in-the-middle effect found that language models can use information less reliably when it appears in the middle of long contexts. A required passage being technically present is not enough if it is buried among twenty distractors.

Inspect results by rank, not just presence. Evidence consistently appearing at position 18 tells you the index works and ranking does not. Evidence missing entirely suggests ingestion, segmentation, vocabulary, or permissions.

Test retrieval variants against the same frozen set:

• Dense semantic search
• Keyword or BM25 search
• Hybrid search
• Query expansion or decomposition
• Metadata and date filters
• Reranking the top candidate set
• Different chunk boundaries and document structure

Change one major variable at a time. If embeddings, chunk size, reranker, and prompt all change together, an improved answer score tells you nothing reusable.

Do not turn chunk overlap into a ritual. RAGChecker found relatively small effects from overlap in its experiments, while retrieval depth and chunk size produced clearer tradeoffs. Your documents may behave differently, but the point stands: measure changes rather than inheriting settings from a tutorial.

Permissions need their own hard gate. Run the same query as an authorized employee, an unauthorized employee, and a user from another tenant. Restricted chunks should never enter the candidate set. Filtering after generation is too late. See our permission-aware agent guide for the architecture behind that test.

Evaluate Generation With Fixed Evidence

Once retrieval is measurable, freeze it. Feed the generator a controlled context pack and test whether it can use that evidence correctly. This removes search variability and reveals generation defects.

Measure at least these behaviors:

• Context use: Did the response include the supported claims relevant to the question?
• Faithfulness: Is each factual claim entailed by the supplied context?
• Unsupported claims: Did the model add facts not present in the evidence?
• Citation correctness: Does each citation support the sentence attached to it?
• Conflict handling: Did the model disclose or escalate incompatible sources?
• Abstention: Did it refuse to invent an answer when evidence was insufficient?
• Instruction compliance: Did it produce the required format, action, and handoff?

Use deterministic checks where possible. Exact IDs, dates, totals, URLs, JSON schemas, and cited source identifiers should be validated in code. Use model graders for semantic judgments, but calibrate them against human reviewers. The RAGAS paper is one useful reference for breaking evaluation into retrieval and generation dimensions without requiring a single reference answer for every case. A grader that approves polished unsupported prose is worse than no grader because it creates false confidence.

Run a small double-review exercise before trusting an automated grader. Have two domain experts independently score 50 to 100 outputs at claim level. Resolve disagreements and rewrite the rubric until reviewers interpret it consistently. Then compare the automated grader against those adjudicated labels. Track false passes on high-risk claims separately from overall agreement.

Citation evaluation deserves special attention. A response can cite a real document that does not support its assertion. Check the relationship between claim and passage, not whether the output contains a link-shaped object. For regulated or consequential workflows, save the quoted evidence span and source version with the trace.

Test robustness by altering the context pack:

1. Correct evidence only
2. Correct evidence plus irrelevant text
3. Correct evidence plus a stale contradiction
4. Relevant evidence with the key claim removed
5. No relevant evidence

A good generator should remain faithful in the first two, follow authority rules in the third, and abstain or escalate in the final two. The NIST AI Risk Management Framework recommends measuring systems in context and managing identified risks through their lifecycle. These controlled context tests turn that principle into an executable release gate.

Connect Component Scores to the Business Result

Retrieval and generation scores are necessary, but buyers pay for completed work. Add a third layer that tests whether the response or action satisfies the workflow.

For a policy assistant, that could mean the correct decision, citation, and escalation. For support, it could mean resolution without an incorrect promise. For a sales agent, it could mean the correct account facts copied into a draft without exposing another customer's data.

Use a failure matrix to connect symptoms to likely owners:

Set release thresholds per segment. A knowledge assistant may tolerate an abstention but not an unsupported legal claim. A low-risk drafting tool may accept minor wording variation. A system that changes records needs stricter action and rollback checks. Our loops and evals guide covers how to turn these checks into ongoing operations.

Track latency and cost beside quality. A reranker that raises claim recall by two points but triples response time may be wrong for frontline support and right for contract review. Report cost per accepted answer or completed workflow, not cost per model call.

Production monitoring should preserve the same stage boundaries. Log the query, user identity, retrieved chunk IDs and ranks, source versions, generated claims, citations, tool actions, grader results, and reviewer correction. Redact sensitive content according to retention policy, but keep enough provenance to reproduce failures. For workflows that change state, pair these RAG checks with the deterministic fixtures and side-effect capture in the AI agent testing sandbox playbook.

A 30-Day RAG Evaluation Playbook

The work can begin without rebuilding the stack.

Week 1: Define the truth. Select one workflow. Gather 100 to 200 real questions. Label required claims, authoritative passages, forbidden sources, user roles, expected actions, and risk. Agree on what "unsupported" and "correct escalation" mean.

Week 2: Baseline retrieval. Run the frozen question set through the current retriever. Score claim recall, context precision, rank, authority, freshness, and permissions. Fix missing ingestion and access errors before tuning search.

Week 3: Baseline generation. Create controlled context variants and score context use, faithfulness, citations, abstention, and conflicts. Calibrate model grading against expert review. Add deterministic validators for structured outputs and important fields.

Week 4: Run targeted experiments. Pick the largest measured failure category. Test one retrieval or generation change at a time. Promote only when the relevant score improves without violating security, cost, or latency gates. Add every confirmed defect to the regression suite.

The output is more than a dashboard. It is a diagnostic loop: failed claim, failed stage, named owner, targeted change, repeatable proof.

How OpenNash Can Help

OpenNash helps teams turn a vague RAG quality problem into a measured repair plan. We map sources and permissions, build the claim-level evaluation set, instrument retrieval and generation separately, calibrate graders, and define release thresholds around the business decision.

An established search or evaluation platform is often enough for standard document search when engineers can configure and operate it. Custom work fits cases where source authority, tenant permissions, workflow actions, and acceptance rules are specific to the company. If nobody owns the authoritative content, fix that governance gap before adding more RAG infrastructure.

The implementation remains owned by the client: evaluation records, source rules, traces, tests, and operating runbook. Book a call to map this scorecard to one RAG workflow and identify which layer is actually failing.