The Memory Problem Behind Long Context
This series examines two different answers to one question: how can a sequence model use information that no longer fits comfortably in its immediate context? Titans: Learning to Memorize at Test Time and Augmenting Language Models with Long-Term Memory (LongMem) are not simply proposals for making a context window larger. They are proposals about where the past should live, how it should be written and retrieved, and how a system should manage the fact that memory is never unlimited. Understanding that distinction gives readers a way to judge long-context claims instead of treating a token count as a guarantee of recall.
A larger window does not solve the memory problem
A Transformer’s attention mechanism is powerful because it can compare the current query with many earlier keys and use the corresponding values. In memory terms, the model keeps explicit key-value associations for the tokens inside its current context. That makes direct dependencies relatively precise: when the relevant passage is present, attention can search it rather than relying on a compressed summary.
The strength has a cost. Standard causal attention becomes more expensive as the sequence grows, and every model still has a practical context boundary. Once an earlier passage falls outside that boundary, the model cannot attend to its original representation in the ordinary way. Increasing the nominal window can postpone the problem, but it does not answer what should happen to the history that remains too long, too expensive, or too distant to keep explicitly available.
Linear attention and recurrent models take the opposite route. They update a fixed vector- or matrix-valued state instead of retaining every past key and value. This is cheaper and can support longer streams, but compression is not free. A very long history must be fitted into a limited state, and additive updates can allow information to accumulate without a reliable policy for what to preserve or discard. The central engineering tension is therefore not simply “short context versus long context.” It is explicit detail versus compressed persistence, with a cost attached to both.
Imagine a model continuing a book after several earlier chapters. A character relationship, a promise made in an earlier scene, or a fact needed for the next chapter may be somewhere outside the immediate window. A dense attention system can preserve the relevant detail if the earlier text remains available, but it must pay to carry and compare a larger body of material. A compressed state can remain available while the text disappears, but its success depends on whether the update rule kept the particular relationship the next prediction needs. “Long context” therefore describes an input arrangement; “long-term memory” describes a policy for preserving and recovering useful information across that arrangement.
That policy also determines what a failure looks like. With ordinary attention, failure may mean that the relevant passage is no longer in the window or that the cost of keeping it there is prohibitive. With compressed recurrence, failure may mean that the information was overwritten, blurred into an inadequate abstraction, or never given a distinct place in the state. The two papers matter because they do not merely stretch the same mechanism. They move the boundary between what is kept explicitly, what is learned into a state, and what is reconstructed when a new query arrives.
Two locations for the past
Titans makes the past part of a changing neural memory. Its long-term memory is a neural module whose parameters are updated while the sequence is being processed. A separate attention pathway supplies short-term memory for the current context, and persistent parameters are intended to hold task-related knowledge that does not depend on the current input. The long-term branch is not just a larger list of old tokens: it learns an abstraction of history in the behavior of a nonlinear function and retrieves that memory through a forward pass.
LongMem places the past somewhere more explicit. Its pretrained backbone remains frozen during memory-augmented adaptation. Earlier segments pass through that backbone, and selected attention key-value representations are stored in an external memory bank. A trainable residual SideNet retrieves relevant entries for a new input and fuses them with the local computation. In this design, the backbone is the stable encoder, the cache owns the stored representations, and the SideNet owns the learned act of reading them.
The distinction is easy to miss because both systems may be described as “remembering” a prior sequence. But they preserve different objects. Titans changes the behavior of a function so that a later query can elicit an abstraction of what came before. LongMem preserves selected representations produced by a stable encoder and gives a later query an indexed route back to them. The first path makes the write rule central: what should alter the memory’s parameters now? The second makes the retrieval interface central: which stored representation is relevant, and at what granularity should it be returned?
That difference changes the important failure questions. For Titans, the questions include whether its update rule learns useful abstractions, whether surprise and forgetting manage finite parameter capacity, and whether the memory can be trained efficiently. For LongMem, the questions include whether the right cached representations are found, whether the cache remains causally and operationally manageable, and whether the reader can combine retrieved history with the current context without stale representations overwhelming it. Both systems still need a short-term route for immediate processing; long-term memory supplements that route rather than making local context irrelevant.
This also changes the meaning of “freshness.” Titans can adapt its contextual memory as new tokens arrive, so a recent sequence can alter the function that will answer later queries. That adaptivity is useful when the stream itself should teach the model what matters, but it makes the update process part of the model’s behavior. LongMem keeps the encoder stable while the cache changes. Stability helps keep old representations compatible with the reader, but the cache still has to decide what to retain and how much of the past can fit. A stable representation is not the same as a complete record, and an adaptive representation is not automatically a faithful one.
The practical question is therefore not which design sounds more like human memory. It is which state transition matches the workload. A stream that rewards learned abstraction may care most about selective updates and forgetting. A language model that must reuse particular demonstrations or retrieve background from earlier chapters may care most about explicit indexing and controllable granularity. Those are different forms of long-term usefulness, even when both are measured by the next-token prediction or answer accuracy.
The questions that decide whether memory works
The two papers can be compared without pretending that their implementations are interchangeable. The first question is where information is stored: in the evolving parameters of a neural memory, or in an external bank of encoded representations. The second is what causes a write. Titans uses an online learning rule tied to the error of an associative key-value prediction; LongMem stores representations produced by its frozen encoder as the cache advances through previous segments.
The third question is how a later input finds the past. Titans queries a learned function. LongMem performs indexed similarity retrieval over cached key-value material and then uses a SideNet to fuse the result. The fourth is how loss is managed. Titans has an adaptive forgetting mechanism; LongMem uses a finite, rolling cache whose contents are updated as new segments arrive. Neither approach can preserve every detail indefinitely, so the system’s policy for relevance matters as much as its nominal capacity.
The fifth question is how long-term information meets short-term processing. Titans experiments with memory as a context, a gated branch, or a layer around attention. LongMem combines local attention and retrieved memory inside a memory-augmented layer. The sixth is what the evidence actually measures. Titans reports experiments spanning language modeling, commonsense reasoning, long-context retrieval, multi-fact reasoning, time series, and DNA modeling. LongMem focuses on long-text language modeling, the ChapterBreak task, memory-augmented in-context learning, and related efficiency measurements. Those scopes are informative but different.
These questions become concrete in the papers’ application choices. LongMem’s many-shot in-context-learning setting loads additional task demonstrations into memory, so the useful past is not just a general summary; particular examples and label-bearing patterns need to be found again. Its long-text setting similarly asks a later passage to benefit from earlier book or paper context. Titans tests a broader sequence-modeling role, including long-context retrieval and reasoning as well as time series and DNA tasks. That breadth makes its memory a candidate for a general sequence backbone, but it also means the reader must look at which component and which benchmark produced each result.
A memory system can also be excellent at one of these jobs and weak at another. A needle-in-a-haystack result asks whether a planted item can be recovered under distractors. It does not by itself show that the system has built a faithful, inspectable model of every relationship in the surrounding document. Conversely, a compact learned abstraction may support a downstream forecast without preserving a verbatim passage that an indexed cache could return. The right evaluation follows the required kind of memory, not only the length of the input.
That last point sets an important boundary. The supplied papers report encouraging results for their own mechanisms, but they do not constitute a controlled Titans-versus-LongMem benchmark. Their model sizes, training regimes, datasets, baselines, and metrics differ. A careful comparison can still ask which memory location fits which constraint; it cannot turn the papers into a universal leaderboard.
Long context becomes useful only when a model can decide what survives the immediate window and can recover it when needed. Titans and LongMem make that decision in different places. To understand the consequences, the next step is to treat a sequence model itself as a system that writes and reads memory, rather than as a black box with a single context-length number.