Two Ways to Extend a Model’s Past

The first part of the comparison has established a useful distinction. Attention keeps explicit key-value material inside a bounded window; recurrent compression carries a smaller state forward; Titans makes that state a nonlinear memory whose parameters learn from the sequence. The judgment now is not whether Titans has “memory,” but what its location and lifecycle solve—and what questions remain open when long-term information is kept somewhere else.

One problem, five boundaries

A long-term memory architecture can be understood through five boundaries that are easy to blur together. The first is the storage boundary: what object contains the past after it leaves the immediate context? In Titans, the contextual past is represented by the changing parameters of a neural memory. The system does not preserve a visible list of all earlier key-value pairs. It preserves the effect of online updates on a function that can later respond to a query.

The second is the write boundary: what event changes that stored object? Titans uses an associative key-value loss and its gradient as a write signal. The current input is not simply appended; it can alter the memory in proportion to how poorly the current association is handled. Momentum-like past surprise lets a related run of inputs influence one another, while a data-dependent decay controls how much of that influence crosses a change of context.

The third is the read boundary: what does a later input actually receive? Titans reads the neural memory through a forward pass without another weight update. The output is what the learned function produces for the query. This is different from asking an external store to return a particular old record. The distinction does not make the Titans read less real, but it does make it a read of an abstraction rather than a direct lookup of preserved text.

The fourth is the loss boundary. Every long-term memory needs a way to keep old information from crowding out new information. Titans applies an adaptive weight-decay gate to the previous memory before adding the new surprise update. The gate can preserve the earlier abstraction, reduce it, or clear it. Forgetting is therefore part of the write transition, not a cleanup step that happens after the memory has already overflowed.

The fifth is the integration boundary: how does long-term information meet the short-term computation that is still handling the current input? Titans keeps short-term attention as a distinct role and explores several ways to connect it with neural memory. The long-term function can be used as contextual input, as a branch combined through a gate, or as a layer around attention. These choices matter because a memory can be accurate in isolation and still be poorly placed for the computation that must use it.

This frame also exposes a shared requirement. Both explicit attention and parameter memory need a policy for relevance. Attention leaves the decision to the current query among the records that remain in the window. Titans moves more of the decision into the write process: the loss, surprise history, and decay determine which aspects of the past alter the function. Neither system escapes selection. They place selection at different points in the lifecycle.