Titans: Learning What to Remember
Titans addresses the hardest part of compressed memory: deciding what an incoming sequence should change. Its proposal is not just to carry a larger recurrent state. It makes the long-term memory itself a neural network whose parameters are updated as the sequence arrives. The useful way to understand Titans is to follow one input through its write rule, the persistence of that write across nearby inputs, the forgetting gate that limits capacity, and the separate read operation that later uses the stored abstraction.
A memory that learns while the sequence arrives
Titans begins with an associative-memory task. Each input representation is projected into a key and a value. The key is the address that a later query will use; the value is the information the memory should return for that address. The neural memory is trained, in its online inner loop, to make the key produce the corresponding value. In simplified form, the memory receives a key, applies its current function to that key, compares the result with the target value, and uses the difference to adjust its parameters.
This turns “memorize at test time” into a specific operation. The system is not changing its long-term memory because an engineer manually inserted a new record. It is applying a learning step to reduce an associative prediction error on the current input. The key-value projections and the rest of the architecture are learned in the outer training loop; the long-term memory’s parameters are the state that changes in the inner loop as new data arrives. Separating these loops is essential. The outer loop teaches the architecture how to use a memory, while the inner loop lets that memory adapt to the particular sequence it is currently seeing.
The error gradient supplies Titans’ write signal. The paper calls this gradient a form of surprise: a larger gradient indicates that the current input is not well handled by the memory’s present behavior, so the input has more influence on the next update. This is a model-defined measure, not a claim that the system understands surprise as a person does. It is also not a guarantee that every semantically important event receives a large score. Its value is operational: the gradient tells the memory how strongly the current key-value association should change the function.
That choice addresses a weakness in a purely additive recurrent write. If every input contributes the same kind of update, the memory has no explicit reason to prioritize a new association over a familiar one. Titans instead lets the current loss determine the direction and magnitude of the correction. An input that the memory already handles well may produce a small change; an input that conflicts with the current abstraction can produce a larger one. The write rule is therefore tied to the memory’s current state rather than to token arrival alone.
A single-step gradient is not enough for a sequence. Suppose a surprising event begins a short episode: the first token changes the memory sharply, but the following tokens may now lie in a region where the gradient becomes small. If Titans reacted only to each token’s momentary gradient, it could preserve the trigger and lose the context that explains what followed. The proposed update therefore carries a second state, described as past surprise, alongside the current or momentary surprise.
The past-surprise state behaves like a momentum term in optimization. A contribution from earlier inputs can continue to affect later writes, while a data-dependent decay controls how long that contribution remains relevant. When the next token appears to belong to the same context, the previous surprise can persist. When the sequence changes context, the decay can reduce the influence of the old episode. This is a temporal interface between inputs: the current token does not merely write its own error; it also decides how much of the recent update history should cross the boundary into the next step.
The distinction matters because “surprise” has two roles in the mechanism. Momentary surprise says what the current input is asking the memory to correct. Past surprise says whether a recent correction should still shape the current write. The first reacts to the present mismatch; the second protects a short episode from being fragmented into unrelated token-level decisions. Titans is consequently not just a gradient update repeated many times. It is a gradient update with a memory of its own write signal.
After the online update, retrieval is deliberately different from writing. Given a query derived from the current input, Titans runs the neural memory forward without adjusting its weights. The forward pass returns the abstraction associated with that query; the memory does not learn again merely because it was read. This separation gives the system a clean lifecycle: the input can cause a parameter update during the write phase, and a later query can use the resulting function during the read phase.
The state diagram also shows why the mechanism should not be summarized as “the model stores surprising tokens.” What is stored is an update to the parameters of a neural function. The update is shaped by a key-value loss, influenced by recent surprise, and moderated by decay. A later read retrieves an output from the resulting function, not necessarily a verbatim copy of the triggering input. That distinction is the basis for calling the memory an abstraction of the past.
Forgetting is capacity management, not erasure
Even a deep neural memory cannot preserve unlimited history with perfect fidelity. Titans adds a separate adaptive forgetting mechanism for this reason. Before the new surprise update is added, the previous memory is multiplied by a factor controlled by the current input. The factor can leave the earlier abstraction mostly intact or reduce it substantially, up to clearing the existing memory in the paper’s formulation. The gate therefore decides how much of the old parameter state should survive the next transition.
This is a different operation from the surprise update. Surprise says how the current input should change the memory. Forgetting says how much of the previous memory should remain available before that change. The two controls can cooperate: a familiar continuation may preserve an existing abstraction and add a small correction, while a strong context shift may decay the old state before learning the new one. Without this distinction, the memory would have to use the same update to both retain useful history and make room for unrelated history.
Consider two adjacent segments in a long stream. The first segment may establish an unusual relationship and produce a large update. If the second segment develops that relationship, the data-dependent decay can allow the earlier surprise to remain influential while the new gradient refines it. If the second segment begins an unrelated topic, the same control can reduce the old contribution before the new topic is written. The gate does not need a hand-authored chapter boundary. It uses the current input to decide whether the previous abstraction should cross the transition. That is the sense in which forgetting is adaptive rather than a fixed schedule.
Momentum and forgetting solve different problems even though both affect what survives. Momentum carries information about the history of the write signal, so a short sequence of related corrections can act as one extended episode. Weight decay acts directly on the memory state before the next correction, controlling how much of the old parameter configuration remains. Momentum without decay could keep an old surprise active for too long; decay without momentum could lose the continuation after the initial trigger. Their reported ablation contributions are therefore meaningful as a design interaction, not just as two unrelated knobs.
The distinction also clarifies what “capacity” means for a parameter memory. A finite neural network can hold many possible associations in its weights, but it cannot keep every episode equally distinct as the stream grows. Selective forgetting does not make the capacity infinite. It gives the memory a policy for spending its capacity: preserve an abstraction when the next inputs appear connected, and release influence when the context changes. A later read succeeds only if that policy kept the relevant relation in a form the query can activate.
The paper relates this weight-decay rule to forgetting gates in modern recurrent models, but its role in Titans is especially clear when the memory is treated as a parameterized function. The gate is not an instruction to delete one named token. It changes the baseline from which the next gradient update operates. What disappears is therefore an earlier parameter configuration or abstraction, not necessarily a discrete record with a visible boundary. This is powerful for compact memory, but it also makes the result less directly inspectable than an external list of stored key-value pairs.
Titans also argues that the memory’s internal structure should be more expressive than a single linear map. A matrix-valued state with one linear transformation treats the key-to-value relationship as an online linear regression problem. That can be useful when the historical dependency is close to linear, but it limits the forms of abstraction the memory can represent. Titans therefore uses a multilayer perceptron as the long-term memory, with deeper versions intended to model nonlinear relationships in the history.
Depth changes both sides of the trade-off. A deeper memory can transform an input through multiple nonlinear stages before producing a retrieved value, which gives it more expressive room than a single matrix update. But those stages also add work. In the reported experiments, increasing memory depth improves perplexity across tested sequence lengths and makes the model more robust as sequences grow, while training throughput decreases roughly linearly with added depth. The paper’s ablations also report positive contributions from deep memory, momentum, and weight decay, but those results belong to the tested models and tasks rather than establishing that every deeper memory is better.
Titans keeps another kind of state separate from this contextual memory. Persistent memory consists of learnable, input-independent parameters intended to encode knowledge about the task itself. The contextual neural memory changes as it processes the current sequence; persistent memory is not supposed to be rewritten by each new context. Short-term attention, contextual long-term memory, and persistent memory thus have different owners and different lifecycles:
- Short-term attention processes the active context and models its direct dependencies.
- Contextual neural memory updates from the sequence and carries an abstraction of its past.
- Persistent memory stores task-related knowledge that does not depend on the current input.
This separation prevents a common category error. A model’s fixed learned weights, its current online memory state, and its attention over recent tokens may all be called “memory,” but they do not change for the same reason or on the same timescale. Titans’ architecture is built around letting those roles coexist rather than forcing one state to serve every purpose.
The design is recurrent in state, but the paper does not accept recurrence as a reason to give up parallel training. Titans divides a sequence into chunks and rewrites the weight-decayed updates using matrix multiplications and sums. The momentum-like surprise recurrence can be computed with a parallel associative scan. These transformations allow accelerator-friendly operations while preserving the sequential meaning of the memory update. The result is a practical compromise: the memory state depends on the sequence order, but the training procedure can expose more parallel work within chunks.
The reported throughput still has conditions. Neural memory is slightly slower than Mamba2 and Gated DeltaNet in the paper’s comparison, which the authors attribute to deeper memory and a more expressive transition as well as optimized baseline kernels. A MAL configuration can be faster when it benefits from optimized sliding-window and full-attention kernels. The paper reports linear scaling with context length for the described neural-memory training procedure, but that does not mean every Titans configuration is cheaper than every attention or recurrent alternative. Depth, kernels, and hardware affect the outcome, and the work’s first version noted that larger-model results were still being finalized.
Titans therefore offers a precise answer to the question of what a compressed memory might learn to keep. It can use associative error as a write signal, carry that signal through a short episode, decay an old abstraction when context changes, and represent the result in a nonlinear parameterized function. The price is that the memory’s contents are shaped by an online learning process and are not simply a searchable archive. The next comparison must ask which of these choices are essential to long-term memory itself and which are consequences of placing memory inside changing parameters.