From Dialogue to Trajectory: Why Agent Memory Needs Causality
Agent memory becomes a different engineering problem when the history contains actions, observations, and tool outputs rather than only conversation. This chapter focuses on that difference: why a memory can retrieve something semantically related yet fail to preserve the causal conditions that make a later answer or action correct. The practical result is a test for memory quality: ask not only whether the retrieved item resembles the question, but whether it contains the state transition and dependencies needed to support what the agent must do next.
A trajectory is a causal record, not a transcript
A dialogue is usually organized around statements. An agent trajectory is organized around interaction with an environment. The agent takes an action, the environment changes—sometimes invisibly—and the agent receives an observation or tool result. That result constrains the next action. A browser click can change which controls are available. A database query can reveal a schema that determines the next query. Picking up an object can make a later action possible, while consuming it can make a previous plan invalid. The meaning of a later turn depends on what happened before it.
This difference matters for memory construction. A conversational summary can often preserve a preference or a conclusion in a compact sentence. A trajectory may require exact identifiers, code, SQL, UI selectors, response values, error messages, locations, inventories, or the order of operations. These details are not decorative. They can be the precondition for a later action or the evidence that a state changed. If a summarizer removes them as low-level noise, it may preserve the topic while destroying the procedure.
AMA-Bench frames the agent’s interaction as a partially observed process. The environment has a state that the agent cannot necessarily see in full. The agent acts, receives an observation, and builds a history from the task instruction, actions, and observations. A memory system turns that history into an external state and later retrieves a query-relevant context. This formal framing makes the hidden dependency explicit: the right answer may depend on a state transition that was never stated as a clean fact in the text.
The benchmark therefore separates four capabilities. Recall asks whether the system can identify temporal or sequential information. Causal inference asks whether it can verify a precondition or dependency between states. State updating asks whether it incorporates new observations and changes the represented state. State abstraction asks whether it can condense the history without losing the precise information needed later. These capabilities overlap, but they fail differently. A system may remember that an agent was working toward an objective while forgetting that a prerequisite failed. It may retrieve the right object but not its current location. It may summarize a plan while preserving an obsolete branch after the environment has changed.
The real-world subset of AMA-Bench uses trajectories from web navigation, open-world tool use, text-to-SQL, software engineering, gaming, and embodied AI. It contains 2,496 expert-curated question-answer pairs. The synthetic subset uses TextWorld and BabyAI to generate trajectories with controllable horizons and verifiable state transitions. The mix matters because the problem is not confined to one interface. Machine-generated representations—tables, JSON, code, HTML, and tool outputs—are part of the evidence an agent must manage, and their structure is often more objective and less redundant than ordinary chat.
Consider a simple causal chain in a text environment. The agent must retrieve a key, use it to unlock a basement, enter the basement, and craft an antidote. A similarity search for “craft antidote” may find the right room and the right object but omit the prerequisite chain. A summary may retain the overall goal while dropping the fact that the door was still locked. Both memories are related to the task. Neither is causally sufficient for the next action. The agent can then repeat an invalid attempt, follow a stale plan, or fail to recover after an irreversible state change.
This is the difference between semantic relevance and causal sufficiency. Relevance asks, “Does this memory concern the same topic?” Sufficiency asks, “Does this evidence justify the answer or enable the next transition?” A memory system designed mainly for similarity can do well on the first question and fail on the second. For an agent, that failure is not merely a bad explanation. It can change the trajectory of actions and make the environment harder to recover from.
The four capabilities form a useful chain. Recall supplies the pieces of an earlier interaction. Causal inference checks whether those pieces explain a dependency. State updating changes the represented world when a new observation arrives. State abstraction decides which details can be compressed and which must remain exact. A failure early in the chain can make later success impossible: no retrieval policy can verify a precondition that construction dropped, and no abstraction can be considered successful if it removes the identifier needed to distinguish two otherwise similar objects. Conversely, a faithful representation can still fail if retrieval selects only a related fragment.
That chain also explains why machine-generated data is unusually unforgiving. In ordinary prose, repeated descriptions may be redundant. In an agent trajectory, a repeated-looking string can be a different table name, file path, selector, object location, or response value. A short command can change the state of a system more than a paragraph of explanation. The memory must therefore decide not only what is important in a linguistic sense, but what is operationally binding. This is why the benchmark treats exact and structured artifacts as part of the memory problem rather than as incidental formatting.
The evidence has a deliberate boundary. The real-world trajectories are observed action-and-observation logs; the benchmark authors do not have the hidden backend state for those tasks. The synthetic environments provide a complementary advantage: their transition rules are programmatically defined, so the supporting turns and state changes can be verified. That combination makes it possible to test both realistic messiness and controlled causal dependencies, but it does not make the benchmark identical to every deployed agent. A system that performs well on a synthetic transition may still need different extraction rules for a web page, a code repository, or an enterprise database.
Preserve relationships, then retrieve beyond similarity
AMA-Bench’s experiments separate where the loss occurs. A raw trajectory can contain the necessary “needle” turns, but a construction method may discard them while compressing the history. A constructed memory can contain the needed information, but the retrieval method may select the wrong neighborhood. The full system can then fail even though the answer was present at the start. In one needle analysis, Mem1 falls from 0.46 when the relevant evidence is available to 0.29 after its construction step; MemoryBank scores 0.27 with constructed memory and 0.26 end to end. HippoRAG2 has 0.37 when its constructed memory includes the needle but falls to 0.21 end to end, illustrating a retrieval-and-answer loss after representation.
These are benchmark-specific measurements, not a universal ranking. Their value is diagnostic. The three-stage comparison asks three separate questions: Was the required state present in the raw trajectory? Did construction preserve it? Did retrieval surface it in a usable form? Without those comparisons, “the model forgot” hides the difference between lossy abstraction and failed search.
AMA-Bench also finds that memory architecture can matter more than the tested change in backbone size. In its cited comparison, moving from an 8B to a 32B backbone improves the average by 0.038, while changing the memory architecture produces a score range of up to 0.45. That does not show that model scale is irrelevant. It shows that, on these tasks, the way the system constructs and retrieves state is a major determinant of performance. A larger model cannot reliably reason over a dependency that its memory representation deleted.
The proposed response is AMA-Agent’s causality graph. Construction works locally on adjacent trajectory material: the previous observation, the action, and the new observation. It extracts environment and object states, represents state relationships as nodes, and creates directed edges for causal dependencies. It also creates undirected association edges for related state or object information. The local structures are then combined into a global graph, whose nodes also receive embeddings.
This design divides two jobs that are often conflated. The graph records a relationship such as “this action changed that state” or “this state enabled that transition.” The embedding supplies a way to discover candidate nodes when the query uses different wording. The graph is not a replacement for similarity search, and similarity is not trusted to recover every dependency on its own. The representation keeps a path that can be traversed after an initial semantic lookup.
Retrieval follows the same division of labor. AMA-Agent first retrieves a small set of nodes by embedding similarity and asks whether the evidence is sufficient. If the query is a straightforward recall question and the returned nodes answer it completely, the process can stop. If the question needs adjacent turns, preconditions, or multi-hop relations, the system can invoke depth-controlled graph traversal. If it needs a count, an exhaustive list, or a pattern across the trajectory, it can invoke keyword or programmatic search rather than estimating from a small similarity-selected subset.
The retrieval loop can be represented as a sequence of state changes and conditional searches:
The tools are selective rather than a permanent replacement for the fast path. In an analysis of 200 sampled questions, graph search was used in 10.0 percent of cases, mainly for multihop or precondition-dependent questions. Keyword search was used in 23.5 percent, typically for counts or patterns spanning the trajectory. Those rates support a conditional design: use embeddings to find likely evidence, then escalate when the question itself signals that similarity-selected fragments may be incomplete.
Under the paper’s Qwen3-32B comparison, AMA-Agent reports an average accuracy of 0.5722 across recall, causal inference, state updating, and state abstraction. The corresponding reported values for HippoRAG2 and MemoRAG are 0.4480 and 0.4606. Removing the causality graph reduces the average to 0.43; removing tool-augmented retrieval reduces it to 0.44. The ablations do not prove that every task needs a graph or tools. They show that both components contribute in the benchmark’s agentic setting: the graph protects relationships during construction, and the tools provide access patterns that ordinary similarity does not cover.
The benchmark also checks whether offline memory questions relate to actual behavior. When the same Qwen3-32B execution agent uses different memory systems in TextWorld and Spider2, QA accuracy and end-to-end success correlate at 0.96 and 0.98 respectively across the six systems in the reported table. The result is encouraging but narrow: two environments and six systems cannot establish a universal calibration law. End-to-end success also combines memory with policy and tool-use errors, which is why targeted QA remains useful for isolating memory quality.
The broader lesson is not that causality graphs solve long-term memory. The graph itself is built through LLM-based semantic abstraction and intentionally discards some low-level artifacts. Its benefit depends on whether the abstraction retains the state, identifier, or dependency the task needs. A graph can preserve a wrong extraction with great consistency, and a tool can search precisely for evidence that was never stored. The design works when the construction contract and retrieval fallback match the workload.
The same caution applies to the tools. Graph traversal is useful when the missing evidence is near a known turn or connected through a prerequisite chain. It is not the natural answer to a question that asks for every occurrence across a long trajectory. Similarity search can locate a likely starting point, but a count over only the retrieved subset will undercount by construction. Conversely, an exact keyword search can find matching strings while missing a synonym or a state relation. The routing decision must be tied to the question’s operation, not added as a decorative layer of complexity.
AMA-Bench’s downstream check is important for that reason. A memory method can score well on a question while still producing an unusable context for an acting agent, or it can look weak on a noisy open-ended answer while preserving the state needed for a successful action. The reported correlations between QA accuracy and end-to-end success are strong in the two tested environments, but the authors still keep the protocols distinct: offline QA makes memory errors easier to isolate, while rollout success tests whether the retrieved state actually changes behavior. Reliable evaluation needs both views when action failure is costly.
That condition is what separates an agent trajectory from a long transcript. For a static fact, semantic similarity may be enough to locate a supporting sentence. For an action sequence, the memory must often preserve what changed, what made the change possible, and what is no longer true. Once those requirements are visible, the next architectural question follows naturally: which memory pipeline stages own construction, storage, retrieval, and maintenance, and what costs are created by each choice?