Choosing a Memory System Under Real Constraints

Choosing a long-term agent memory system is not a contest to find the highest score on one benchmark. It is a decision about which updates must remain recoverable, how often state changes, how quickly a read must reflect those changes, and what construction and retrieval work the service can afford. This chapter turns the earlier pipeline and failure modes into a conditional choice: first define the memory question and operating constraints, then select the simplest design that preserves the required evidence and test it under the workload that will actually run.

Start with the question the memory must answer

The first criterion is semantic, not technological. A system that mainly answers “What is true now?” has a different requirement from one that must answer “What was true after the third revision?”, “Which action caused this state?”, or “How many times did this pattern occur across the whole trajectory?” Current-state recall, historical lookback, causal verification, and exact aggregation should not be collapsed into one accuracy score.

For recall-heavy work over relatively stable text, a full context or a simple lexical or dense index may be a reasonable starting point. The choice is conditional. Full context keeps the raw evidence directly available while the history fits a practical input budget and the query volume does not make repeated prefill too costly. Flat retrieval keeps construction cheap and predictable, but it relies more heavily on matching the query to the right chunk. A system that only needs recent, explicit facts may not justify an expensive graph or agentic write path.

The decision changes when the query names a prior state, a relationship, or a set of evidence. Historical questions require version identity and provenance. Causal questions require links between actions and consequences. Counting and ordering questions require coverage beyond the top few similar fragments. Exact identifiers, code, SQL, selectors, and state values may need to remain verbatim even when other prose can be summarized. In these cases, a richer representation, structured metadata, graph traversal, or exact search can be justified—not because it is inherently superior, but because the query operation cannot be performed reliably from one semantically similar passage.

Evidence fidelity is the next criterion. Ask what must survive construction. Is it enough to retain a paraphrase, or must the system preserve dates, revision identity, commands, errors, locations, preconditions, or the relation between an old and new value? A consolidating store may reduce redundancy and make current-state retrieval fast, but it needs a rule for what happens to the prior state. An append-oriented store may preserve both versions, but it needs a way to identify which is current and to prevent retrieval from treating contradictory entries as interchangeable. A graph may expose relations, but its extraction process must preserve the relations the workload actually uses.

The construction–serve–accuracy frontier makes the trade visible. The systems-characterization study reports that BM25 is fast to construct and performs strongly on a macro-average across a suite that includes many recall-heavy tasks. The same paper cautions that lexical retrieval has weaknesses for paraphrase matching, multihop reasoning, and temporal inference—the kinds of questions that motivate richer structure. Structure-augmented and agentic systems can offer more expressive access or mutation, but they pay additional construction, retrieval, or control-flow cost.