Overview

This chapter reads the original RAG paper’s results across open-domain QA, MS-MARCO, Jeopardy question generation, FEVER, and retrieval ablations. RAG reports strong task-specific comparisons: it reaches a reported state of the art on four QA datasets, improves the BART baseline on MS-MARCO’s Rouge-L and BLEU-1, and receives more favorable human factuality and specificity judgments than BART in the Jeopardy comparison. FEVER also separates label accuracy from article retrieval. These results show useful retrieval–generation behavior, not universal factuality: metrics measure different properties, and article overlap is not sentence-level entailment.

Learning outcomes

On the paper’s four open-domain QA tasks, RAG reports state-of-the-art comparisons, but exact match does not reveal evidence use or claim support.
RAG-Sequence exceeds BART on the reported MS-MARCO Rouge-L and BLEU-1 scores, while missing or mismatched Wikipedia evidence limits interpretation.
FEVER retrieval overlap, human preferences, and distinct-trigram ratios measure different properties; none alone establishes sentence-level provenance or truth.

Keywords

RAGBenchmarksFEVERMS-MARCOFactualityMetrics

What the Experiments Actually Show: Answers, Questions, and Claims

The experiments in the original RAG paper test a specific proposition: connecting a generator to retrieved text can improve knowledge-intensive behavior, but the improvement depends on what the task asks the system to do and on how the result is measured. This chapter reads the evidence across question answering, free-form generation, question generation, fact verification, and retrieval ablations. The goal is to separate observed results from interpretation, so that a benchmark gain is not mistaken for a universal guarantee of factuality or grounding.

The experimental portfolio matters because “better with retrieval” can mean several different things. A system may find the right passage, produce a correct short answer, write a more specific sentence, generate a more diverse formulation, or classify a claim correctly. Those capabilities overlap, but no single metric measures all of them. The paper’s comparisons are strongest when the task, baseline, and evidence path are kept visible.

The benchmark results favor flexible answers, with important qualifications

The paper evaluates RAG on four open-domain question-answering datasets: Natural Questions, TriviaQA, WebQuestions, and CuratedTrec. Questions and answers are treated as input-output text pairs, so RAG is allowed to generate an answer rather than extract a span from a retrieved passage. The paper reports a new state of the art on all four tasks under its comparison conventions. For TriviaQA, however, it reports two setups: the standard open-domain test arrangement and a TQA-Wiki test set used for comparison with a different line of work. That split distinction is part of the result, not a footnote that can be removed when quoting the score.

On the reported test scores, RAG-Sequence reaches 44.5 exact match on Natural Questions, 56.8 on the standard TriviaQA setup and 68.0 on the TQA-Wiki setup, 45.2 on WebQuestions, and 52.2 on CuratedTrec. RAG-Token reaches 44.1, 55.2 and 66.1, 45.5, and 50.0 respectively. Exact match is a strict measure: the generated answer must match an accepted answer form. It is useful for short answers, but it does not describe how a longer answer used evidence or whether a near-match was substantively correct.

The comparison supports the paper’s central argument about generation versus extraction. An extractive reader is constrained by the spans it can locate in its retrieved documents. RAG can generate an answer from a passage that contains clues without containing the answer verbatim. The authors report a particularly revealing subset on Natural Questions: when the correct answer was absent from every retrieved document, RAG still achieved 11.8% accuracy, while an extractive model would score 0% in those cases. This demonstrates a capability of the hybrid system—the generator can combine a clue with parametric knowledge or complete an answer—but it does not demonstrate that RAG can generally answer safely without evidence.

The result also changes how “retrieval failure” should be interpreted. For an extractive system, failure to retrieve the answer span is close to a hard stop. For RAG, missing wording does not necessarily end the computation because the generator can use related passages and its learned associations. That increases the system’s range of possible answers, but it also makes the basis of a correct answer less uniform.

Published by DeepMox

FAQ

What did RAG report on open-domain question answering?

The paper reports a new state of the art for RAG on Natural Questions, TriviaQA, WebQuestions, and CuratedTrec under its comparison conventions. TriviaQA includes both a standard setup and a TQA-Wiki comparison set, and exact match is limited to accepted answer forms.

What does the FEVER experiment show about RAG?

FEVER shows that RAG can use retrieved context for claim classification, while article overlap is not sentence-level entailment. The paper reports 72.5% three-way and 89.5% two-way accuracy, with a gold article among the top ten in 90% of cases.

Does a higher RAG benchmark score prove factuality?

No. Exact match, Rouge-L, BLEU-1, Q-BLEU-1, human preferences, and distinct-trigram ratios measure different outcomes rather than complete evidence support. None alone establishes claim-level provenance or truth.

Previous8 minHow RAG Learns Evidence

Next10 minWhat RAG Changes—and Cannot Guarantee

RAG Results: What the Experiments Show

What the Experiments Actually Show: Answers, Questions, and Claims

The benchmark results favor flexible answers, with important qualifications

FAQ