Overview

This chapter compares RAG-Sequence and RAG-Token as two ways to assign retrieved evidence during generation. RAG-Sequence keeps one latent document for a complete candidate answer and uses document-specific decoding; RAG-Token marginalizes the top-K documents at each token and can use a standard beam transition. Sequence-level conditioning favors coherence and a simpler document unit for inspection but can cost more and limit composition. Token-level conditioning supports local contributions from several passages but makes provenance and consistency harder to follow. The paper’s results vary by task, so neither formulation is universally superior.

Learning outcomes

RAG-Sequence keeps one document in view for a whole candidate answer, favoring document-level coherence at the cost of flexibility.
RAG-Token mixes document-conditioned next-token distributions, enabling multi-passage composition but making consistency and attribution harder to summarize.
The paper’s results are task-dependent: RAG-Token leads on Jeopardy generation, while RAG-Sequence leads on several reported QA and MS-MARCO comparisons.

Keywords

RAG-SequenceRAG-TokenDecodingCoherenceCompositionProvenance

One Passage or Many? RAG-Sequence and RAG-Token

This chapter compares the two original RAG formulations by one decisive question: how much of an answer should be guided by the same retrieved document? RAG-Sequence keeps one document responsible for a whole generated sequence, while RAG-Token can let different documents influence different tokens. Understanding that choice gives readers a way to judge the trade-off between document-level coherence and multi-document composition, and to see why the choice changes decoding as well as model behavior.

The difference is not a cosmetic variation in notation. It changes what counts as a plausible explanation for an answer. A short response may depend on one passage that contains all the necessary context. A longer response may require several pieces of information or may benefit from a passage that supplies an initial clue while the generator completes familiar wording. The two formulations allocate responsibility differently before the first token is produced.

RAG-Sequence asks one document to carry the answer

RAG-Sequence treats the retrieved document as one latent variable for the complete target sequence. For every candidate in the top-K set, the generator calculates the probability of the whole answer while conditioning on that document. The system then weights each sequence probability by the retriever’s probability for the document and sums the alternatives. In plain language, each candidate gets to make a case for the entire answer, and RAG combines those document-level cases.

This can be expressed as a simple comparison among complete candidates. Passage A is evaluated with the full output sequence, passage B is evaluated with that same sequence, and so on. The model does not ask which passage supports the first half and then silently switch for the second half. It asks how well each passage-conditioned run accounts for the answer as a whole, then preserves the alternatives through the marginal probability. The document is therefore a unit of explanation, even though the final result can reflect more than one candidate.

This assumption creates a form of coherence. If one passage contains the relevant context, all tokens in the answer are generated with that passage available. The document does not change halfway through the sequence simply because a later token would be more probable under another candidate. That can make the document-level interpretation easier to explain: one can ask which retrieved passage best accounts for the answer as a whole.

Coherence is an assumption, not a guarantee. An answer can legitimately draw on several passages, especially when the question asks for multiple facts. A single candidate may contain only one part of what is needed. RAG-Sequence can still use the generator’s parametric memory to complete or transform the passage, but it does not give the model a new retrieved document for a later part of the answer. The same-document constraint may therefore favor consistency at the cost of flexibility.

The constraint also affects search at test time. A usual autoregressive beam search works with a per-token transition probability: at each step, it extends partial answers according to their next-token scores.

Published by DeepMox

FAQ

When is RAG-Sequence a better fit?

RAG-Sequence is a better fit when one passage can provide coherent context for the complete response and document-level consistency matters. Its same-document constraint can reduce flexibility when an answer needs several sources.

What does RAG-Token change at each token?

RAG-Token allows the document influencing one token’s probability to differ from the document influencing another token’s probability. This supports local contributions from several passages but can combine incompatible scopes.

Why is RAG-Sequence decoding more involved?

RAG-Sequence decoding is more involved because its probability is defined over complete document-conditioned sequences, not one ordinary next-token transition. The paper uses document-specific beams with Thorough or cheaper Fast Decoding.

Previous7 minRAG: Who Knows What?

Next8 minHow RAG Learns Evidence

RAG-Sequence vs RAG-Token

One Passage or Many? RAG-Sequence and RAG-Token

RAG-Sequence asks one document to carry the answer

FAQ