From Query to Sentence: How Retrieved Text Enters Generation
This chapter explains the handoff inside RAG: how the retriever’s passages become usable context for BART, the sequence-to-sequence generator, and how that context changes the probability of an answer one token at a time. The specific focus is the boundary between finding text and writing text. Following that boundary lets readers distinguish what RAG retrieves, what the generator computes, and why the system keeps several possible passages in play instead of treating the highest-ranked passage as a hard fact.
A retrieved passage is not yet an answer. It is an input to another model that must decide what to say, in what order, and with what wording. That distinction is easy to miss because the passage and the generated sentence may discuss the same subject. But they are different objects with different owners. The retriever selects candidate text; BART uses the original input, the candidate text, and its previous output tokens to predict what comes next.
The generator receives two kinds of context
In the RAG model described by the paper, the generator is BART-large, a pre-trained encoder-decoder. “Encoder-decoder” means that one part reads an input context and another part produces an output sequence. RAG does not require BART to learn language from scratch each time a query is asked. Its parameters already contain learned patterns for understanding context and producing text. Those parameters are the system’s parametric memory.
The retrieved passage supplies a second, temporary context. RAG concatenates the original input with the retrieved content before passing it to BART. For a question, the combined input therefore contains both the question and one candidate passage. The generator reads that combination and predicts a distribution over possible next tokens. A token may be a whole short word, part of a word, punctuation, or another unit used by the model’s vocabulary. After one token is selected, it becomes part of the previous-output context used to predict the next token.
Concatenation is a small implementation choice with a large conceptual consequence. The generator is not handed a detached search result that a separate rule engine must interpret. The query and passage enter the same conditioning context, so BART can use the wording of the question to decide which parts of the passage matter and can use the passage to alter the continuation it would otherwise prefer. This is a shared interface, not a one-way transfer of a finished answer. The retriever supplies material; the generator still owns the transformation from material to output.
The flow is autoregressive. At the beginning, the generator has the original input and retrieved passage but no answer tokens. It predicts a first token. At the next step, it has the same conditioning context plus that first token. At the following step, it has the same context plus the first two tokens, and so on until it produces an end marker or reaches the decoding limit. The generator is not merely scoring a finished passage. It is repeatedly deciding how the answer should unfold.
That repeated decision creates a useful separation of responsibilities:
- The original input states the task the system is being asked to perform.
- The retrieved passage supplies explicit text that may help with the task.
- The generator parameters supply language competence and learned associations.
- The previous output tokens establish what the answer has already committed to.
These inputs can reinforce one another, but none should be confused with another. A query can be clear while retrieval is poor. A passage can be relevant while the generator fails to use it. A fluent continuation can be produced from the parameters even when the passage is missing an answer. The output is the result of their interaction, not a direct readout of the retrieval score.
The original input remains important after retrieval. It tells the generator what relationship to extract or express. A passage that mentions a person may be useful for a question asking about that person’s work, but the generator still has to connect the passage to the question. Conversely, a passage may contain several facts and the question may require only one. Concatenation makes the material available; it does not specify the final answer by itself.
The generator also does not lose its pre-trained language ability when explicit text is added. BART was trained as a denoising sequence-to-sequence model, so it already has a mechanism for turning context into a coherent output. RAG changes the context supplied for the current input. It does not replace BART with a passage copier. This is why the same interface can produce a brief output or a longer explanation: the generator repeatedly applies the same next-token operation until the target sequence is complete. What changes is the number and content of decisions, not the basic handoff.
A short output and a long output exercise the interface differently. For a short label, only a few next-token decisions may be needed, so a relevant passage can shift the choice quickly. For a longer answer, early choices become part of the context for later choices. A small misunderstanding at the beginning can influence the rest of the sequence, while a useful passage can continue to shape the answer across many steps. The sequence format makes RAG flexible across output lengths, but it also means that context use is a continuing process rather than a one-time citation lookup.
A simple way to state the causal chain is: retrieval changes the evidence available to the generator, and the generator changes that evidence into language. If the retrieved text includes a useful clue, it can shift the generator toward a more specific continuation. If the text is irrelevant or misleading, it can shift the distribution in an unhelpful direction. If it is absent, the generator may still rely on its parameters, but that fallback is not equivalent to consulting the missing source.
The important design question is how to represent uncertainty about the passage. A conventional retrieval pipeline might choose the top result, place it in a prompt, and proceed as though that choice were settled. The RAG formulation instead keeps a top-K set and lets the candidate documents participate in a combined probability calculation.
The sequence diagram makes the order visible. Search happens before the generator can use the external text. The generator is evaluated in the context of candidate passages. The marginalizer combines the resulting predictions before the next output decision is finalized. In the paper’s formal description, the retriever probability weights each document-conditioned generator probability. The diagram represents that calculation as a separate step so that the reader can see that “retrieval score” and “next-token probability” are not the same quantity.
For each candidate passage, BART can assign a different probability to the next token. One passage may make a specific noun likely; another may make a different wording likely; the original input and the already generated tokens constrain both. RAG does not simply add the text strings together and ask BART to choose a quotation. It evaluates how the generator behaves under each candidate context, then combines those model outputs. That is why the document choice sits inside the probability calculation rather than outside it as a preliminary yes-or-no filter.
Why RAG keeps multiple passages in play
RAG treats the retrieved document as a latent variable. In ordinary language, that means the system does not have to announce one document as the uniquely correct explanation before generating. Instead, it considers the candidate documents in the top-K set and marginalizes over them. Marginalization is a way of adding the weighted alternatives together: each passage contributes according to how likely the retriever considers it and how well the generator predicts the target in that context.
Suppose several passages are plausible for a question. One may state the key fact directly. Another may contain a definition that clarifies the question. A third may use different wording that helps the generator recognize the relationship being asked about. A hard top-one choice would discard those alternatives before generation. A marginal calculation preserves their influence in the output distribution. This does not mean every candidate is equally useful or that combining them always improves the answer. It means the architecture does not force an early, irreversible commitment to a single passage.
The distinction between a weighted alternative and a verified source is essential. A high retriever probability says that a passage is favored by the query-document matching model. A high generator probability says that, given the input, passage, and prior tokens, a particular next token is a likely continuation. RAG combines these model judgments. Neither judgment independently establishes that the passage is true or that the resulting sentence is entailed by it.
The top-K restriction is the practical approximation that keeps the calculation manageable. The full space of possible documents is much larger than the few passages retrieved for one input. RAG sums over the selected candidates rather than every document in the index. This leaves out passages that the retriever ranked below the cutoff, including a passage that might have been useful but failed to surface. The approximation is therefore both a computational boundary and an evidence boundary. Increasing K can expose the generator to more alternatives, but it also increases the work required to evaluate them and can introduce more irrelevant material.
The model’s uncertainty is not confined to the retrieval stage. Even if the correct passage is among the candidates, BART must interpret it in relation to the input and the partial answer. The generator may favor a smooth continuation that is not fully supported, or it may use a relevant clue to complete a fact from its parametric memory. This is part of RAG’s design value: generation can go beyond exact quotation and combine information. It is also why retrieved context should be treated as evidence available to a prediction, not as a guarantee attached to every predicted token.
The same distinction explains why a retrieved passage can be helpful even when it does not contain the final wording. A question may use one expression while the passage uses a related expression. The generator’s learned language ability can bridge that difference. It may also combine a passage with knowledge encoded in its parameters. In an extractive reader, a missing verbatim answer can be a hard barrier. In RAG, the passage can contribute a clue that changes the generator’s distribution without supplying a ready-made span. The benefit comes from the interface between retrieval and generation, not from retrieval alone.
That flexibility creates a failure path worth keeping in view. If the needed passage is not in the top-K set, the generator cannot use it as current explicit context. It may still produce an answer from its parameters, but the answer then has a different basis than one grounded in the missing passage. If the passage is present but ambiguous, the generator must resolve the ambiguity. If several passages disagree, marginalization does not itself decide which source deserves trust; it only combines model-weighted alternatives. The architecture preserves possibilities, but it does not supply an independent authority for resolving them.
The treatment of documents also prepares the distinction between the two RAG formulations. In one formulation, the same document is used to calculate the probability of every token in the sequence. In the other, the model can use a different candidate document for each token’s calculation. Both begin with the same retriever-generator interface described here, but they make different assumptions about how evidence supports an answer. One favors document-level consistency; the other permits finer-grained composition.
For now, the main conclusion is narrower and more useful: retrieved knowledge enters RAG through conditioning and probability, not through a direct copy operation. The retriever decides which text is eligible to influence the answer. BART decides how the input, that text, its learned parameters, and its previous commitments become a sequence. Marginalization delays a hard commitment among candidate passages, but the top-K cutoff and the generator’s own judgments remain part of the result.
Once this handoff is clear, the next question is about responsibility across the sequence. Should one retrieved passage remain in charge from the first token to the last, or should different passages guide different pieces of an answer? That choice determines how RAG handles coherence, composition, and the visibility of evidence.