The Missing Half of a Language Model: Why RAG Retrieves Before It Speaks
Retrieval-augmented generation, or RAG, is the subject of this series: a model design that gives a text generator access to an explicit collection of retrieved passages while it writes an answer. The focus is not retrieval by itself, and not language generation by itself, but the boundary between them—how a question becomes searchable evidence, how that evidence changes a generator’s choices, and where the arrangement stops being trustworthy. Understanding that boundary gives readers a way to judge RAG systems without treating a fluent answer as proof that the system knew or verified what it said.
The idea begins with a tension inside modern language models. A pre-trained model can learn a great deal about the world from the text used to train it. That knowledge is stored in its parameters—the numerical values adjusted during training. The result can be remarkably capable: the model can answer, continue, summarize, and translate without looking anything up at the moment of use. Its memory is always present and its language skill is available for many tasks.
But that convenience makes the memory difficult to manage. If a fact changes, changing the model’s behavior may require more training rather than editing a plainly visible record. When the model produces an answer, its parameters do not provide a straightforward, human-readable account of which sentence supplied each detail. And a model can generate a plausible statement even when its internal associations are incomplete or wrong. For knowledge-intensive tasks—questions whose answers depend on facts a person could not reasonably be expected to recall unaided—those are not minor inconveniences. They concern whether the system can use current evidence, whether someone can inspect that evidence, and whether the answer can be checked.
When knowledge is inside the model
It helps to separate two capabilities that are often bundled together under the word “knowledge.” The first is the ability to produce language: to form grammatical sentences, follow a prompt, combine concepts, and complete a partially written thought. The second is access to a particular fact at the right moment. A parametric model can support both, but it supports them through the same learned machinery. That makes the boundary between “I can express this” and “I have a reliable basis for this” hard to see from the output alone.
Parameters are not a shelf of labeled entries that a person can open and revise one at a time. They are learned numerical relationships distributed across a very large network. Those relationships can make the model good at recognizing that two ideas belong together, choosing a likely continuation, or paraphrasing a familiar fact. They do not naturally expose a tidy record saying which training sentence caused a particular token. This is why a model can be knowledgeable in practice while remaining difficult to audit as a knowledge store. The problem is not simply that the model has too little information; it is that the information is embedded in a form optimized for prediction rather than direct maintenance.
RAG does not throw away this learned capability. In the paper’s design, the generator is BART, a pre-trained sequence-to-sequence model. Its parameters remain the part of the system that knows how to turn an input and a context into a coherent sequence. What changes is that the generator receives another kind of memory: raw text stored outside its parameters. The external text can be selected for a particular input, inspected by a person, and replaced as the underlying collection changes.
This creates a division of labor rather than a replacement. The external memory can supply a passage about the middle ear, a clue about an author’s works, or a statement relevant to a fact-checking claim. The generator can use that passage together with the original question and its own learned language ability. A retrieved passage may give the generator a precise expression that it would not have selected on its own; conversely, the generator may be able to combine a clue with what it already knows or produce a useful answer when the wording in the passage does not exactly match the answer.
The distinction matters because an explicit passage is easier to inspect than a hidden parameter association, but visibility is not the same as authority. RAG’s retrieved text is context supplied to a generator. It is not automatically a verified fact, and the generator is not reduced to copying it. A passage can be irrelevant, incomplete, stale, or biased. The generator can also make a mistake while using relevant material. The purpose of retrieval, then, is to create a better information path into generation—not to turn generation into a database lookup.
There are two corresponding places where an answer can go wrong. The retriever may fail to place the useful passage among the candidates, so the generator never receives the needed evidence. Or the retriever may find a relevant passage and the generator may still misread it, combine it incorrectly, or add an unsupported detail. A hybrid design makes these stages visible enough to analyze, but it does not make either stage infallible. That distinction will matter later when the paper’s benchmark gains are compared with its retrieval and provenance limits.
RAG’s two memories in one view
The original RAG model makes this relationship concrete. For an input such as a question, a retriever scores passages in a document index and returns a small set of high-scoring candidates. The generator then conditions on the input and those passages while producing the output. Instead of declaring one candidate to be the answer source, the model treats the retrieved document as a latent variable and combines predictions associated with the top candidates. “Latent” here means that the document choice participates in the probability calculation, even though the final text is what the system emits.
The diagram’s two arrows into the generator express the central idea. One path carries the original input and retrieved text from the current search. The other represents the generator’s learned parameters, which were formed before this particular question was asked. RAG’s behavior emerges from their interaction. Retrieval determines which explicit material is available; generation determines how that material and the input become a sequence of tokens. The final output is therefore neither a pure quotation from the index nor a pure recollection from the parameters.
The top-K step is important even at this high level. The system does not pass an entire encyclopedia into the generator. It first narrows a large text collection to a manageable candidate set, then lets the candidates contribute according to the retriever’s scores and the generator’s predictions. A high retrieval score means that a passage is favored by the matching mechanism; it does not mean the passage has been checked by a separate fact verifier. Likewise, a passage’s presence in the context does not mean every word of the final answer came from it. The interface creates a chain that can be inspected—query, candidates, context, output—but the chain still contains judgments and approximations.
This is also why RAG is more than a generic instruction to “look up information first.” The paper joins a dense neural retriever to a pre-trained generator and fine-tunes them as a probabilistic model. The retriever’s ranking matters because it controls which passages enter the candidate set. The generator’s use of those passages matters because relevant text can still be ignored, misunderstood, or combined with other learned associations. The interface between the two is the subject worth examining.
What this series will make possible
The practical promise is easiest to state in three parts. First, a searchable text memory can be expanded or revised more directly than a model’s parameters. The paper demonstrates this with two Wikipedia snapshots: replacing the non-parametric index changed which world-leader answers the same RAG model produced, without retraining the generator. That result shows that external memory can carry some of the burden of keeping knowledge aligned with a changing world. It does not mean that replacing an index edits every association already stored in the generator, or that a newer snapshot is automatically more accurate.
The experiment makes the difference between “accessible knowledge” and “complete model knowledge” concrete. The model was queried about 82 leaders whose answers changed between the December 2016 and December 2018 snapshots. When the snapshot matched the period of the question, the reported accuracies were substantially higher than when the index came from the wrong period. The result is not a general freshness guarantee, but it demonstrates a useful architectural lever: the text memory can be changed independently of the generator and can influence the answers that follow.
Second, the retrieved passages give a person something concrete to inspect. Seeing candidate text is a meaningful improvement over having only an opaque parameter state. But the improvement is an affordance, not a complete provenance system. RAG can combine several passages with its parametric memory, and the paper’s retrieval analysis measures whether a gold article appears among retrieved articles rather than proving that every generated claim is entailed by a particular sentence. Inspection helps a reviewer ask better questions; it does not answer those questions automatically.
Third, generation preserves flexibility. In the paper’s open-domain question-answering experiments, the authors argue that documents containing clues can contribute to a correct answer even when the answer is not written verbatim in any one retrieved passage. That is a real difference from an extractive system, which is limited to spans it can locate. It is also a source of responsibility: the generator can go beyond the text, so a reader must ask whether that extra step is supported.
Those benefits lead to the central question for the rest of this series: how can retrieved knowledge enter a generative model in a way that improves access, revision, and inspectability without making the retrieved text—or the fluent output—seem more certain than the evidence warrants? To answer it, the next step is to look closely at the external memory itself: how documents become searchable passages, how a query is matched to them, and why those design choices determine what the generator is able to see.