In brief
How do you evaluate a RAG system?
Build a test set of realistic questions with expected answers and supporting sources, then measure three stages separately: retrieval (was the right content found), grounding (did the answer stay faithful to it) and answer quality (was the final answer correct, complete and useful). Automate what can be automated, calibrate any model-based grading against human reviewers and rerun the evaluation whenever something changes.
Retrieval-augmented generation is the most common architecture for business knowledge assistants. A question arrives, relevant passages are retrieved from company documents, and a language model writes an answer based on them. When the answer is wrong, teams often respond by changing the prompt or switching models. Frequently the real problem is earlier in the chain.
Three places a RAG system can fail
| Stage | Failure looks like | Typical causes |
|---|---|---|
| Retrieval | The right passage is not among the results | Poor chunking, weak embeddings, missing documents, no keyword matching for codes or names |
| Grounding | The answer adds claims the sources do not support | Prompt allows model knowledge to fill gaps, too many irrelevant passages, conflicting documents |
| Answer quality | The answer is faithful but incomplete, unclear or unhelpful | Instructions unclear about format, missing context about the user, sources themselves outdated |
Measuring each stage tells you where to spend effort. There is little point tuning prompts if the relevant passage was never retrieved.
Building the test set
The test set is the most valuable asset in a RAG project. It should be created with people who know the subject and the users.
- Collect real questions from support tickets, search logs, emails or interviews rather than inventing idealised ones.
- For each question, record the correct answer and the document passages that support it.
- Include difficult cases: questions spanning several documents, questions using internal abbreviations, questions whose answer changed recently and questions the system should decline because no source answers them.
- Tag questions by topic and type so results can be broken down.
- Keep a held-out portion that is not used while tuning, to avoid overfitting prompts to the test set.
Measuring retrieval
Retrieval can usually be evaluated automatically, because the test set records which passages should be found. Useful measures include whether at least one supporting passage appears in the top results, and how highly the first relevant passage is ranked. Break results down by question type: retrieval that works well for policy questions may fail for product codes, which often need keyword or hybrid search alongside embeddings.
Measuring grounding
Grounding, sometimes called faithfulness, asks whether every claim in the answer is supported by the retrieved content. It is the main defence against fabricated answers. It can be assessed by breaking an answer into individual claims and checking each against the sources, either by trained reviewers or by a model instructed to do so. A system that refuses to answer when evidence is missing should score well here, which is exactly the behaviour you want.
Measuring answer quality
Correctness and completeness are best judged against the reference answers in the test set. Usefulness depends on the audience: an answer for a customer needs different detail and tone from one for an engineer. Define a short rubric with subject experts, for example a score for correctness, completeness, clarity and appropriate citation, and apply it consistently.
Using models to grade, carefully
Human review does not scale to every change, so many teams use a language model as a grader. This works when it is calibrated. Take a sample of answers, have experts grade them, then have the model grade the same answers and compare. If agreement is poor, refine the grading instructions or keep human review for that dimension. Recheck calibration periodically, especially after changing the grading model.
Making evaluation continuous
RAG systems change constantly: documents are added, chunking is adjusted, prompts are edited and providers update models. Each change can improve one area while quietly damaging another. Run the evaluation automatically on every change and block releases when key scores fall below agreed thresholds. In production, sample real conversations, review them against the same rubric and add difficult new questions to the test set.
Sources and further reading
About the author: Dr. RohitCo-Founder, Director and Chief Technology Officer, AARHIT Systems. Writes on technology strategy, AI architecture, agents, evaluation and AI governance. More articles by Dr. Rohit