The Memory Pipeline: Where State Is Lost and Cost Is Paid

Long-term agent memory is not a single store that a model consults when it forgets. It is a pipeline that ingests interaction, constructs a representation, persists it, retrieves part of it, and maintains it as new information arrives. This chapter maps those stages and the four broad design paradigms around them, then follows the state and resource costs through a long-lived workload. The result is a way to see why a fast query can hide expensive construction, a stale write, or a maintenance problem.

One memory system, seven boundaries

The first boundary is ingestion: what portion of the interaction stream enters the memory process and in what unit. A system may process individual turns, fixed-size chunks, sliding windows, or complete sessions. That choice is not merely a batching detail. It determines how often the system can react to a change and how much surrounding context is available when it decides what the change means. A small unit may notice a local update quickly but lack the context needed to resolve it. A large unit offers more context but can make each update more expensive and less frequent.

Next comes memory construction, the write-path transformation from raw interactions into persistent records. Construction may be absent, as in a full-history prompt; deterministic, as in chunking and indexing; LLM-mediated, as in extracting facts, summaries, triples, or structured records; or agentic, as in allowing an LLM to decide when to write, what to change, and which memory operation to use. Construction is where the system chooses what survives. A later retriever cannot recover a relationship that construction removed, or an old value whose version identity was compressed away.

Storage determines which operators the read path can use. A store may retain raw prompt text, an inverted lexical index, dense vectors, a graph, structured records, or several of these views together. A raw archive can preserve detail but may require more work at query time. A compact fact store can make retrieval fast but depends on the extraction and conflict rules that produced it. A graph can expose relationships and paths, but it adds construction and traversal work. The storage choice is therefore a statement about state ownership: the store owns the durable representation, while the active model sees only what later retrieval makes available.

Retrieval selects records for the current query. The simplest forms are lexical or dense top-k search. Richer forms combine sparse and dense search, reranking, graph expansion, metadata filters, or several passes of LLM planning. Agentic systems can turn retrieval into a control loop in which the model decides whether to search again or whether the evidence is sufficient. That flexibility can improve access to historical or causal information, but it also makes work and latency less predictable.

Prompt assembly is the boundary between the memory system and the answering model. Retrieved records are serialized into the active context, where they compete with the query and any current interaction. A memory system can have excellent stored evidence yet assemble it in a confusing, too-long, or incomplete form. Generation then produces the answer or selects an action. The answering model is important, but it is the last stage in a chain whose earlier decisions determine what the model can see.

The seventh stage is maintenance. A long-lived store needs some policy for deduplication, consolidation, conflict resolution, forgetting, compression, or re-embedding.