One Mental Model for RAG: Who Knows What, and When?

The RAG model becomes easiest to judge when a single answer is followed from the user’s input to the passages made available to the generator and then to the words that appear on screen. This chapter connects those stages into one operating picture. The practical question is not merely whether RAG retrieved something, but which part of the system made each consequential decision and whether the available evidence is strong enough to support the final claim.

A useful way to follow the system is to ask “who knows what, and when?” The text index knows only what was persisted in its source collection and represented for search. The retriever knows which candidates best match the current input according to its learned scoring function. The generator knows how to turn context into a sequence and also carries associations in its parameters. The person reading the output must still decide whether the result is adequately supported. These are different forms of access, and collapsing them into one idea of “the model knowing” hides the system’s actual boundaries.

One answer crosses several decision boundaries

Imagine a user asks a question that requires a particular fact and a concise explanation. The first decision is a ranking decision. The query encoder turns the input into the representation used against the fixed document index. Search returns a small set of candidate passages, each with a score or probability weight. At this point RAG has not written an answer. It has selected the text that the next component is allowed to consider.

That distinction remains important even when the top passage looks obviously relevant. Retrieval answers a question of the form, “Which stored passages resemble or fit this input?” It does not answer, “What sentence should be asserted?” A passage can match because it contains the right entity, a related phrase, or a useful clue. The generator must still interpret the relationship between the question and the passage. It may need to select one detail from several, express the detail in different words, or combine a clue with learned language knowledge.

The second decision is a generation decision. BART receives the original input together with a candidate passage and predicts the next token. The previously generated tokens become part of the context for the next step. This process repeats until the sequence ends. The generator therefore owns the transformation from available context to output language. Retrieval controls eligibility; generation controls expression and continuation.

The difference can be seen by considering two hypothetical failures. If the retriever returns passages about the wrong subject, the generator may produce a fluent answer from irrelevant context. If the retriever returns a useful passage but BART misreads its scope, the failure occurs after retrieval. The final sentence may look similar in both cases, but the remedy would differ: the first points toward corpus or query-matching problems, while the second points toward context use or generation. A system-level explanation needs to preserve that distinction.

A query also has a changing relationship with the system over time. Before the query arrives, the index stores the external memory and the generator parameters store learned associations. During retrieval, the query encoder selects candidates from the existing index. During generation, the retrieved text becomes temporary context alongside the original input and the generator’s learned state.