What a Sequence Model Can Keep
To understand why Titans and LongMem place long-term memory in different locations, we first need a common operating model. A sequence model does not “remember” in one universal way: it writes information into some state, later reads from that state, and pays a cost for whatever the state cannot preserve. This chapter separates those three operations in attention and recurrent compression so the later comparison rests on mechanisms rather than labels.
Attention: a bounded associative memory
A Transformer’s attention can be understood as a content-addressed store. Each input representation is projected into a key, which describes how the item can be found, and a value, which contains what should be returned if it is found. A later representation produces a query. The model compares that query with earlier keys, turns the similarities into weights, and combines the corresponding values. The result is not a single summary of the past. It is a set of explicit associations that can be queried differently by different later tokens.
Causal attention imposes an important ownership rule: a position can read the current position and earlier positions, but not future positions. Within the active context, the attention state therefore grows as new key-value pairs are appended. At one step, the model may ask for a name; at another, it may ask for a location or a causal relationship. Because the earlier values remain available, the same passage can answer different queries without being rewritten into one fixed summary first.
That flexibility explains why attention models direct dependencies well. If an earlier sentence says that a person made a promise and a later sentence asks what they must do, the later query can compare itself with the keys of the relevant words and draw on their values. The mechanism does not need to know in advance which relationship will matter. It keeps a collection of possible associations and lets the current query select among them.
The cost is tied to that explicitness. As the context grows, more queries must be compared with more keys, and the attention computation and memory requirements grow quadratically with sequence length in the standard formulation. Efficient implementations can reduce overhead, and sparse or approximate variants can limit which positions interact, but the basic design question remains: how much past can be kept as individually addressable material before the cost becomes impractical?
The context boundary is therefore more than a user-interface limit. It is a boundary on the memory that this particular read operation can see. If a passage has been pushed outside the active window, ordinary attention cannot recover its original key and value. The model may still produce a plausible continuation from its learned parameters or from a summary supplied in the current input, but that is a different source of information. It is no longer a direct read from the earlier passage.
This distinction matters in long documents. A later chapter may need a small fact hidden in an earlier chapter, but the model cannot know at the beginning which facts will become relevant. Keeping every earlier token explicit preserves optionality, yet it makes every later step pay for a larger store. Trimming the context reduces cost, but it also turns the decision about what to retain into an external policy. Attention is excellent at reading what is present; it does not by itself decide how to preserve what has left the window.
Recurrent compression: cheaper state, harder retention
Recurrent models make the opposite choice. Instead of retaining every earlier key and value, they maintain a state that is updated when a new input arrives. In the simplest operational description, the system performs a write
a new state = update(previous state, current input)
and then a read
the output = read(new state, current input).
The equations are simple, but they expose the central trade-off. A fixed state can be carried forward without reopening the entire history, so inference can remain efficient as the stream grows. The same fixed capacity must nevertheless represent everything that may matter later. Compression makes the state cheaper to move, but it also makes the write rule responsible for deciding what survives.
Linear attention provides a useful bridge between the two views. With an identity-like kernel, a key and value can be accumulated into a matrix-valued state: the state at time t is the previous state plus a contribution built from the current key and value, and a query reads from that matrix. This preserves an associative-memory interpretation while allowing the computation to be arranged recurrently. It is still a compressed state, however. The model no longer keeps every pair as an independent record; it folds them into one object whose capacity and structure are fixed by the architecture.
That additive update creates a predictable failure path. If every new association is added without a mechanism for erasing or correcting older ones, the state can accumulate conflicting or irrelevant information. A long sequence may then overflow its useful capacity even if the numerical state itself continues to exist. A later query receives an aggregate that contains too much history in the wrong form, rather than a clean record of the item it needs. Reducing the computation from quadratic to linear has not removed the memory problem; it has moved the problem into state management.
The type of state matters as well. The papers distinguish vector-valued recurrent memory from matrix-valued memory used by linear attention. A matrix can preserve richer key-value associations than a single vector under the same broad idea of compression, but it still imposes a structure on the dependencies it can represent. A simple matrix update behaves like an online linear regression objective. If the useful relationship in the history is not well described by that form, the state may be efficient but inadequate. This is why “recurrent” is not a complete explanation of a model’s memory behavior: the structure of the state determines what kinds of abstraction are available.
A short timeline makes the distinction clearer. At time t, attention can append the new key and value to the active collection, then let the current query read from that collection. At time t + 1, the earlier record is still available if it remains inside the causal window. A recurrent system instead uses the new input to transform the state that arrived from time t − 1, and the output is read from the transformed state. The next step does not reopen the earlier input; it receives only the state that the previous write produced. The architecture has traded direct access to the earlier record for a compact object that is easy to carry forward.
That trade is about more than speed. It changes where responsibility sits. In attention, the representation of each active token remains available and the query does much of the selection work. In a recurrent model, the update rule must anticipate future queries well enough to encode the right distinctions before those queries arrive. A state can be numerically stable and still be semantically wrong for the next task. Conversely, an attention window can contain the needed passage and still fail if the query does not assign it enough weight. Storage and retrieval are coupled in their outcomes, but they are separate places where a design can succeed or fail.
Nor does a lower asymptotic cost remove all engineering choices. A recurrent update still has a state dimension, a transition rule, and a read operation. Increasing the state can improve representational capacity while increasing the work per step. A more expressive transition can capture richer relationships while making training or inference less efficient. The papers’ long-term memory proposals are meaningful in this space because they ask whether the state should be a simple fixed object, a deeper learned function, or an explicit bank coupled to a reader. The answer cannot be inferred from the word “linear” or “recurrent” alone.
Titans and LongMem make different responses to this bottleneck. Titans asks whether the compressed state can itself be a deeper nonlinear function whose parameters learn from the incoming stream, with an update rule that includes retention and forgetting. LongMem keeps selected key-value representations outside the local window and adds a retrieval path that can bring relevant records back when needed. Those are not merely two implementation details. One changes the function that represents the past; the other preserves an indexed collection of representations and changes how the current model reads them.
The diagram’s two paths share the same abstract read/write loop, but their failure points differ. Attention loses access when a needed record is outside the window or when preserving all records becomes too expensive. A compressed state can remain available indefinitely in principle, but the information may have been merged, overwritten, or represented in a form that cannot answer the later query. A useful long-term memory system must address both survival and addressability: information must remain in some usable form, and a later input must have a way to recover the part that matters.
Three design questions hidden inside “memory”
The first question is memory structure. What is the state made of: explicit key-value records, a vector, a matrix, or the parameters of a nonlinear function? Structure determines what can be represented without loss and what kinds of relationships a read can express. It also determines whether a later system can search individual records or must ask a learned function to reconstruct an abstraction.
The second question is memory update. Does each input append information, add a correction, alter parameters through an online learning step, or trigger a forgetting gate? An update is not just a performance optimization. It is a policy about relevance. If the update is too weak, the memory remains tied to old assumptions. If it is too aggressive, a new or surprising input can damage useful context. The design must also say who owns the state and when the transition occurs: per token, per segment, or during a separate adaptation phase.
The third question is memory retrieval. How does the current input ask for the past? Explicit attention compares a query with available keys. A recurrent state applies a read function to a compressed object. An external cache can use an index, but it must choose a granularity and a candidate set. A system may therefore fail even when the desired information was technically stored: the query may not address it, the representation may be too coarse, or the retrieval may return a distractor.
These questions lead to four practical tests for Titans and LongMem. Capacity management asks what happens when the stream is longer than the memory can faithfully represent. Effective recall asks whether the right information can be recovered under distractors, not merely whether the architecture accepts a long input. Update cost asks how much work is required to write or refresh memory as the sequence grows. Local integration asks whether retrieved or reconstructed history helps the current short-term computation, rather than competing with it or overwhelming it.
The tests are related but not interchangeable. A design may have a large nominal capacity and poor effective recall. It may retrieve accurately but spend too much time or memory doing so. It may learn a useful abstraction for forecasting while being unable to reproduce a particular passage. Or it may preserve explicit representations but use the wrong retrieval granularity for a task that depends on a small label or a multi-step relationship. A memory architecture is therefore best understood as a set of coupled state transitions, not as a number printed next to the context window.
With this vocabulary, the two subjects can be compared without flattening them. Titans will be examined as a system that learns how to write, retain, and read a nonlinear memory while processing the sequence. LongMem will be examined as a system that stores encoded representations in a rolling bank and trains a separate reader to retrieve them. The next question is what happens when the memory state itself becomes a learner: can it decide which events deserve to change the parameters that will answer future queries?