The External Memory: Turning Documents into Searchable Evidence
This chapter examines the external-memory side of the RAG model introduced by Lewis and colleagues: how a collection of text becomes searchable evidence, how a query is matched to that collection, and why those choices determine what the generator can see. The practical value is a concrete mental model of the retrieval boundary. Instead of imagining that RAG can consult an unlimited store of facts, readers will be able to trace which passages are available, who selected them, and which limitations are inherited from the stored corpus.
The phrase “external memory” can sound more mysterious than it is. In this system, it means text that is kept outside the generator’s learned parameters and prepared for search. A query does not cause the model to reread an entire encyclopedia. It is converted into a representation that can be compared with representations prepared for many passages. A small, ranked set is then passed onward. The usefulness of that set depends on the content of the collection, the way it was divided, and the rule used to measure relevance.
A corpus becomes a memory through preparation
The paper’s main experiments use a December 2018 dump of Wikipedia. Each article is divided into disjoint chunks of 100 words, producing 21 million documents for retrieval. Here, “document” does not necessarily mean a complete Wikipedia article. It is the chunk that the index can return as a unit of context. This choice makes a large collection manageable and gives the generator passages of a bounded size, but it also creates the first important boundary: information that is separated across chunks may not arrive together in one retrieval result.
The chunks are not stored only as ordinary text. A document encoder converts each one into a dense vector, a list of numbers intended to capture features useful for matching it with a query. The vectors are placed in a FAISS index. The paper uses a Hierarchical Navigable Small World approximation to speed up search, allowing the system to look for high-scoring matches without comparing a query exhaustively with every stored vector. The search structure is therefore part of the memory’s behavior, even though it does not contain new facts by itself. It determines how a large set of representations can be searched at practical speed.
That preparation changes the meaning of “the model has access to Wikipedia.” The generator does not have direct, guaranteed access to every sentence in the dump at every step. It has access to the passages that survive the indexing and retrieval process for the current input. The corpus supplies the possible evidence; the search operation supplies a candidate subset. If a relevant fact is absent from the snapshot, divided awkwardly, represented poorly, or not ranked highly enough, the generator may never receive it as context.
Chunking creates a second boundary beyond simple retrieval failure. A factual statement is rarely just an isolated noun or verb; it may depend on a date, an exception, a comparison, or the sentence that identifies what a pronoun refers to. A 100-word chunk can preserve those relationships when they fall close together, but it cannot guarantee that all qualifying context stays within the same unit. The index can return a passage that looks relevant because it contains the right entity while omitting the relation that makes the entity answer the question. Chunking is therefore part of evidence design: it affects not only whether a topic is found, but whether the retrieved text is sufficient to interpret it.
The physical scale also matters because memory is an operating resource, not just a conceptual label. The paper reports that storing the Wikipedia index vectors on a CPU requires roughly 100 GB of CPU memory, and that FAISS compression can reduce this requirement to 36 GB. Its appendix describes 21 million 728-dimensional vectors and notes that 8-bit storage can help manage the footprint. These figures belong to the paper’s particular index and should not be mistaken for a universal RAG requirement. They do show, however, that “just add a knowledge base” hides a substantial persistence and search layer.
That layer has a lifecycle. First, a source collection is selected and divided into passages. The document encoder maps those passages into vectors. The index persists the relationship between each vector and its text. At query time, the system searches the vectors, then recovers the corresponding text passages for the generator. If the source snapshot changes, the index must be rebuilt or otherwise replaced so that the searchable representations match the new text. The generator’s ability to use the memory is therefore bounded by a durable artifact that has its own version, coverage, storage cost, and maintenance needs.
This lifecycle also separates two kinds of maintenance. Editing or replacing text changes what the index can offer, while changing the document encoder changes the geometry in which passages are represented and may require recomputing the stored vectors. The paper avoids that cost during fine-tuning by keeping the document side fixed. That makes the index easier to use as a stable base for experiments, but it also means the representation layer is not automatically refreshed whenever the downstream task changes. External memory is editable, yet its editability still depends on keeping text, vectors, and search assumptions aligned.
The result is a useful distinction between corpus quality and search quality. A poor corpus can be searched efficiently and still fail to support an answer. A rich corpus can contain the needed fact and still fail if the indexing or ranking process does not surface it. RAG’s external memory is thus not a passive warehouse. It is a designed interface between a source collection and a later prediction.
DPR turns a query into a search decision
The RAG retriever follows Dense Passage Retrieval, or DPR, a bi-encoder design. “Bi-encoder” means that the query and each passage are encoded separately. A BERT-base query encoder maps the input question or claim to a query representation, written in the paper as q(x). A BERT-base document encoder maps a passage to a document representation, written as d(z). The system compares those representations through their inner product: broadly, it favors passages whose vectors point in a compatible direction with the query vector.
This separation is what makes a large index possible. The document vectors can be computed ahead of time and stored. When a new query arrives, only its query representation needs to be produced before the search begins. The system then looks for passages with the highest retrieval scores. In the paper’s formulation, the retriever probability is proportional to the exponential of the document-query inner product. Readers do not need the equation to operate the idea: the retriever converts semantic compatibility into a ranked distribution over candidate passages, and higher-scoring candidates receive more weight when RAG combines their contributions.
Finding the highest inner products is a Maximum Inner Product Search problem, usually shortened to MIPS. The paper uses an approximate solution so the search can run in sub-linear time rather than scanning every passage in full. “Approximate” matters. It describes how the search is performed, not whether the underlying corpus is true. A fast ranking mechanism can make a wrong or incomplete candidate available just as efficiently as it can make a useful one. Search efficiency and evidentiary reliability are related system concerns, but they are not the same property.
The diagram below shows the ownership boundary. The document side prepares and stores the representation of the passages. The query side receives the current input and produces the representation used for the lookup. MIPS connects the two by returning a top-K set. Only after that selection does the raw text re-enter the path toward generation.
The paper initializes this retriever from a pre-trained DPR bi-encoder trained to retrieve passages containing answers to TriviaQA and Natural Questions. That history matters when interpreting the claim that RAG learns retrieval from its task. The RAG fine-tuning setup does not directly label which passage should be retrieved for every new example, but the initial retriever is not an untrained blank slate. It carries a retrieval capability learned with supervision on those QA tasks.
During RAG’s own training, the document encoder and its index remain fixed. The authors explain that updating the document encoder would require periodically rebuilding the index, a costly operation. The query encoder is fine-tuned along with the BART generator. This is a clean state-ownership decision: the persisted document representations stay stable while the query side learns how to ask that existing index better for the downstream task. The arrangement reduces training cost, but it also means that the external memory is not continuously reshaped to match every change in the query encoder. The query representation moves within a geometry whose document side is held in place.
That fixed-versus-changing boundary affects what “learning to retrieve” means. The system can learn that a particular kind of question should point toward a different region of the existing index. It cannot, through this fine-tuning step alone, add a new passage to the corpus or make a missing fact appear in the stored vectors. Improvement in the query representation helps only when the needed material already exists somewhere in the searchable memory. This is why retrieval gains and corpus updates are separate operations: one changes how the system asks, while the other changes what it has to ask.
That choice helps explain why retrieval can improve without becoming a separate, fully supervised pipeline. The generator’s answer loss can reward query representations that bring useful passages into the candidate set. But a useful answer does not necessarily identify one uniquely correct passage. Several passages may provide partial clues, and a generator may supply language or connections from its parametric memory. RAG therefore trains the usefulness of the interface at the level of the final task while leaving the evidence assignment partly latent.
It also explains why a dense retriever should not be treated as universally superior to simpler alternatives. The paper compares its learned dense retriever with BM25, a word-overlap-based method. BM25 performs best for FEVER in the reported comparison, which the authors suggest may be because FEVER claims are heavily centered on named entities and therefore suit lexical overlap. Differentiable retrieval improves the other tasks, especially open-domain question answering. The lesson is not that one search geometry wins in all settings. It is that the retriever must match the kind of relationship the task requires: semantic paraphrase in some questions, exact entity or term alignment in others.
A query can also be difficult for reasons that no encoder alone can solve. It may be ambiguous, assume context not present in its wording, or ask for a fact that falls outside the snapshot. The retrieved passage can contain the right entity but not the needed relation. A high score only says that the passage was favored by the matching function. It does not say that the passage answers the question, that its claims are current, or that the generator will use it correctly. Those are later stages of the RAG path.
The external memory therefore ends at a precise handoff. The corpus owner determines which text is persisted and how it is chunked. The document encoder and index determine how that text is represented and searched. The query encoder turns the current input into a request against that fixed geometry. MIPS returns candidate passages, but it does not write the answer. The next boundary is the generator: how it receives the original input and these passages, and how it turns a set of possible evidence sources into a probability distribution over sentences.