What Is Answer Completeness in RAG Evaluation?

What is answer completeness in RAG evaluation? It is the measure of whether a generated answer covers everything a good answer should contain. Most tools treat it as recall: list the key facts in a reference answer, check which ones show up in the response, and report the share that made it, so a score of 0.5 means half of the expected information is missing. Some judges skip the reference and instead ask whether the answer handled every part of the question.

Completeness is the other half of faithfulness. Faithfulness asks whether everything the model said is backed by the retrieved text, which is precision. Completeness asks whether the model said everything it should have, which is recall. An answer that says "I could not find that" is perfectly faithful and completely incomplete. This guide explains the two ways completeness is defined, how Microsoft Foundry, Amazon Bedrock, Ragas and the research frameworks behind them compute it, and how to read the number. If you want the precision side first, read what is faithfulness in RAG evaluation; for all RAG metrics side by side, see RAG evaluation metrics.

Answer completeness in one example

Say a user asks your support bot about the return policy, and your reference answer contains three facts:

1. Items can be returned within 30 days.
2. You need the original receipt.
3. Gift items get store credit only.

The bot answers: "You can return items within 30 days if you have your receipt." Both of its claims are true and supported by the policy page, so faithfulness is perfect. But it covered two of the three expected facts, so a recall style completeness score is 2 out of 3, about 0.67. The customer with a gift item walks away with the wrong idea, and only the completeness metric notices.

That is the whole recipe in most tools: break the expected answer into small facts, check each one against the response, and divide the facts found by the facts expected.

Two definitions you will run into

Vendors use the word completeness for two related checks, and it pays to know which one a dashboard is showing.

Reference based completeness (recall). You supply a ground truth answer. The scorer checks how much of it the response covers. Microsoft states this plainly in its RAG evaluators documentation: groundedness "focuses on the precision aspect of the response," while response completeness "focuses on the recall aspect," meaning the response "doesn't miss critical information compared to the expected response or ground truth."

Question coverage completeness. The judge reads the question and the answer and asks whether every part of the request was handled. Amazon Bedrock's built in Completeness metric is described as measuring "how well the responses answer and resolve all aspects of the questions." A reference answer helps but is optional.

The first is more precise because there is a fixed answer key. The second works on traffic where nobody wrote a reference, but it only knows what the question asked for, not what your documents say a full answer contains. A question like "what is your return policy?" has no visible parts to count, so a coverage judge cannot tell that the gift rule was left out.

How each tool names and scores it

Microsoft Foundry: Response Completeness

In Microsoft's evaluation SDK the metric is ResponseCompletenessEvaluator. It takes only two inputs, the response and the ground_truth, plus a judge model deployment. Like Foundry's other RAG evaluators it returns a score from 1 (very poor) to 5 (excellent), with a default pass threshold of 3, and a written reason. Microsoft pairs it explicitly with Groundedness so you get both precision and recall for the same answer. It is still marked preview.

Amazon Bedrock: Completeness

Bedrock offers Completeness in its retrieve and generate evaluation jobs, next to Correctness, Helpfulness, Faithfulness and citation metrics. Its judge prompt, published in the Bedrock evaluator prompt docs, tells the model to identify "any crucial information or key points that are present in the reference response but missing from the candidate response," and to focus on main ideas rather than minor details. The judge picks one of five labels, from "Not at all" to "Yes," and Bedrock normalizes the result to a 0 to 1 average on the report card. The Bedrock results guide gives an example of a report showing Completeness at 0.82, with most responses landing between 0.7 and 0.8.

Bedrock also has a retrieval side cousin, Context coverage, which checks whether the retrieved passages contained all the information in the ground truth. Keep those two apart: one grades the retriever, the other grades the answer.

Ragas: FactualCorrectness in recall mode

Ragas has no metric literally called answer completeness, but its FactualCorrectness metric computes it. An LLM breaks both the response and the reference into claims, then natural language inference decides which claims overlap. Reference claims missing from the response count as false negatives, and recall is true positives divided by true positives plus false negatives. That is answer completeness. The default mode is f1, which blends precision and recall, so set mode="recall" if completeness is what you want to track.

Two settings, atomicity and coverage, control how finely sentences are split into claims. Higher values produce more, smaller claims, which gives a completeness score more resolution on long answers. Change them between runs and your scores will shift even if the app did not.

Ragas's answer relevancy metric also says it "penalizes answers that are incomplete," but it works by generating questions from the answer and comparing embeddings, so it is a rough signal at best. Use it to catch answers that drift off the question, not to measure missing facts.

DeepEval and TruLens

DeepEval has a metric called contextual recall, and the name causes confusion. It breaks the expected_output into statements and checks whether each one can be attributed to the retrieval_context. That makes it a retriever metric: it tells you whether the facts were available, not whether the answer used them. TruLens centers its RAG triad on context relevance, groundedness and answer relevance, with no dedicated completeness score. In both tools the usual route is a custom LLM judge with a completeness rubric, which our LLM as a judge guide covers.

Where the idea comes from: nuggets and claims

Completeness is older than RAG. The TREC Question Answering Track introduced nugget evaluation in 2003: assessors list "information nuggets," the atomic facts a good answer should include, mark some as vital and the rest as okay, then check each system answer for each nugget.

The TREC 2024 RAG Track brought the method back with LLMs. In The Great Nugget Recall, Pradeep and colleagues at the University of Waterloo describe AutoNuggetizer, which uses GPT-4o to create nuggets from relevant passages and to label each nugget in an answer as supported, partially supported or not supported. The track's primary score, Vital Strict, is the share of vital nuggets that are fully supported. Across 301 queries with answers capped at 400 words, the fully automatic scores ranked systems much like human assessors did (a run level Kendall's tau of 0.887 against human edited nuggets). But per topic agreement was much weaker, and the authors warn that the framework is not yet good enough for debugging individual answers. They also found the LLM assessor stricter than the human assessors from NIST, the agency that runs TREC.

Amazon's RAGChecker paper uses the same claim level logic. It extracts claims from both the response and the ground truth answer, then defines overall recall as the share of ground truth claims the response entails, with precision and F1 alongside. RAGChecker also separates two causes of an incomplete answer: claim recall, which asks whether the retrieved chunks contained the ground truth claims, and context utilization, which asks how many of those retrieved facts the generator actually used.

What the research says about completeness scores

RAGChecker's authors had annotators compare 280 pairs of answers on correctness, completeness and overall quality, then checked which automatic metrics agreed with them. Three results stand out for anyone choosing a metric:

• Faithfulness says almost nothing about completeness. Ragas Faithfulness correlated with human completeness judgments at a Pearson of 4.90 (on a 0 to 100 scale) and TruLens Groundedness at 14.01. You cannot infer one from the other.
• Claim recall tracks humans best. RAGChecker's completeness metric reached 60.67, against 71.91 for agreement between two human annotators. Ragas Answer Similarity, an embedding comparison with the reference, came second at 53.16 Pearson, and ROUGE-L reached 47.88.
• Real systems leave a lot out. Averaged over 10 domains, the best of eight tested pipelines (an E5-Mistral retriever with GPT-4) had an overall recall of 53.0, so it covered roughly half of the ground truth claims. Retrieving 20 chunks instead of 5 raised claim recall from 61.5 to 77.6.

The authors also found context utilization correlated strongly with overall F1, which matches what practitioners see: low completeness is often a generator problem, not only a retrieval one.

How to read an answer completeness score

1. Check retrieval first. If context recall (or Bedrock's Context coverage) is low, the facts never reached the model, and no prompt change will fix completeness. Tune retrieval depth, chunking and the embedding model.
2. Good retrieval with low completeness is a generator problem. The facts were in the context and the model dropped them. Common causes are answer length limits, prompts that ask for brevity, and models that stop after the first matching passage.
3. Always pair it with faithfulness. Completeness alone rewards long answers that pad in every possible fact, true or not. Faithfulness alone rewards short, evasive answers. Read the two together.
4. Audit your reference answers. A reference with six claims where two are trivia will punish answers that sensibly skip them. TREC marks nuggets vital or okay for this reason, and scoring only vital facts is a good habit.
5. Read averages, not single scores. With three reference facts, one answer can only score 0, 0.33, 0.67 or 1. The TREC results show judges agree with humans far better across a whole run than on any one answer.
6. Keep the judge fixed. Claim splitting and matching are model calls. Pin the judge model, prompt and Ragas atomicity and coverage settings, and record them with each run.

When to use answer completeness

Use it when leaving something out causes harm: policy and benefits questions, medical or legal summaries, troubleshooting steps, and any question with several conditions or exceptions. Run the reference based version on a fixed test set in CI, since it needs ground truth answers, and a question coverage judge on sampled production traces that you capture in a tool such as Langfuse. Our guide on how to evaluate LLM applications explains how to wire both loops, and LLM evaluation metrics shows where claim based scores sit next to BLEU, ROUGE and judge scores. To compare libraries that implement these checks, see best LLM evaluation frameworks.

FAQ

What is answer completeness in RAG evaluation, in one sentence?

It is the share of the information a good answer should contain, usually the key facts in a reference answer, that the generated response actually covers, so it tells you what the model left out.

What is the difference between answer completeness and faithfulness?

Faithfulness is precision: are the claims in the answer supported by the retrieved context? Completeness is recall: does the answer include all the facts it should? Microsoft's documentation describes groundedness and response completeness in exactly those terms, and RAGChecker's study found faithfulness scores barely correlate with human completeness judgments.

Does answer completeness need ground truth?

The most reliable version does. Microsoft's Response Completeness evaluator and Ragas FactualCorrectness both require a reference answer. Amazon Bedrock's Completeness judge can run without one by checking whether every part of the question was addressed, but that cannot catch facts the question did not ask for by name.

Is context recall the same as answer completeness?

No. Context recall, in Ragas and DeepEval, checks whether the retrieved passages contain the facts in the reference answer. It grades the retriever. Answer completeness checks whether the final response contains those facts. You can have perfect context recall and a poor completeness score if the model ignored what it was given.

What is the basic purpose of RAG evaluation?

To find out whether a retrieval augmented system fetches the right information and turns it into answers that are supported, relevant and complete, and to tell which step failed when they are not. Completeness covers the last of those, the part most dashboards skip.

What is a good answer completeness score?

There is no universal cutoff. Microsoft's default pass mark is 3 on its 1 to 5 scale, which is a starting point, not a target. Label 50 to 100 answers by hand as complete enough or not, then pick the score that separates them, and treat changes between runs as more meaningful than any single value.