Two Paths, One Design Question

Titans and LongMem answer the same broad need without implementing the same kind of memory. Titans lets a neural memory change its parameters as a sequence arrives, so the past becomes a learned nonlinear function. LongMem keeps a pretrained language model stable, stores selected key-value representations in a rolling bank, and trains a separate reader to retrieve and fuse them. The final judgment is therefore conditional: choose where the past should live by asking what must survive, how it should change, how it will be found, and what cost and governance the workload can accept.

Memory is a lifecycle, not a context-length number

A context window tells us how much material a model can expose to its immediate attention mechanism. It does not tell us what happens when the useful past is longer than that window. Long-term memory begins when the system must make a sequence of state transitions: store something, update or preserve it, retrieve it later, integrate it with current processing, and manage what is lost when capacity or relevance changes.

The storage transition is the first decision. Titans stores contextual history in the evolving parameters of a deep neural memory. The representation is an abstraction produced by online learning, not a visible archive of the original key-value pairs. LongMem stores selected key-value representations in an external cache created by a frozen backbone. The cache is explicit enough to index and search, but it is still selective: chunks can be evicted, representations can be coarse, and retrieval can return the wrong material.

The write transition is different as well. Titans uses the gradient of an associative key-value loss as a model-defined surprise signal. Past surprise carries a recent episode across adjacent inputs, while adaptive weight decay controls how much of the old parameter state survives before the next update. LongMem writes representations as previous segments pass through the fixed encoder and updates the rolling cache as newer segments arrive. Its main write decision is not a gradient change to the backbone’s memory behavior; it is which encoded history remains in the finite bank.

The read transition exposes the sharpest contrast. Titans runs its neural memory forward on a query after the online update. The answer is an abstraction generated by the current parameter state. LongMem retrieves indexed chunks, expands them into token-level key-value pairs, computes a memory-attention result, and mixes it with local attention through a head-wise gate. The answer is assembled from selected representations and a learned fusion decision.

Forgetting and freshness are also part of the lifecycle. Titans can decay the previous parameter state when the current input indicates that an earlier context should no longer dominate. This supports adaptation, but it also means a poorly timed update can influence later reads until decay or later learning changes the state. LongMem keeps the backbone representation space stable during memory-augmented adaptation, which addresses one source of representation staleness, but it still has to evict old entries and keep retrieval and fusion useful. Stability and adaptability solve different freshness problems.

Both paths retain short-term processing. Titans combines its neural memory with attention through designs that treat memory as context, a gated branch, or a layer. LongMem combines local attention with retrieved memory in a memory-augmented layer. Neither system says that long-term memory should replace the immediate context. The long-term path supplies information that the local path cannot see; the local path supplies precise, current dependencies that a compressed or retrieved memory may not contain.