Overview
This series explains why a longer context window is not the same as long-term memory, using Titans and LongMem as contrasting designs. It moves from attention and recurrent compression to the state lifecycle: where the past is stored, what writes it, how later inputs read it, how capacity is managed, and how memory meets short-term processing. Titans updates a nonlinear neural memory online with associative error, past surprise, and adaptive decay, then reads the resulting abstraction alongside attention. LongMem freezes its backbone, stores selected key-value representations in a finite rolling cache, and trains a SideNet to retrieve chunks and gate them with local attention. Reported results span different tasks and setups, so they support conditional design judgments rather than a universal ranking. The practical choice depends on whether the workload needs a learned evolving abstraction or explicit searchable context, plus acceptable recall, freshness, latency, cost, and governance boundaries.
Learning outcomes
- A longer context window delays memory loss but does not define what survives beyond the active attention window.
- Titans learns a nonlinear abstraction online, using associative surprise, past-surprise momentum, and adaptive forgetting to manage finite parameter memory.
- LongMem keeps selected key-value representations in a finite rolling cache and uses chunk retrieval plus gated SideNet fusion to reuse them.
- The papers support workload-specific trade-offs, not a controlled universal ranking: test recall, freshness, integration, cost, and governance.
Keywords
Preview every chapter · Unlock full text, audio, and video from chapter 4
PlansFAQ
How is Titans different from LongMem?
Titans learns an online nonlinear parameter memory, while LongMem stores selected key-value representations in a finite cache and trains a SideNet to retrieve and fuse them. Their papers use different models, tasks, and metrics, so this is not a controlled ranking.
When is adaptive parameter memory useful?
Adaptive parameter memory is a relevant path when a workload rewards an evolving, learned abstraction of a stream rather than a verbatim record. It trades direct inspectability for compact nonlinear state, so update behavior, forgetting, and effective recall must be tested.
What does LongMem actually retrieve?
LongMem retrieves selected contiguous chunks of cached key-value representations, then expands them to token-level pairs for memory attention. Mean-pooled chunk keys guide search, and a head-wise gate mixes the memory result with local attention.
What should be tested before deploying long-term model memory?
Before deployment, test effective recall under distractors, multi-fact reasoning, update and freshness behavior, full-path cost, and governance under distribution shift. The supplied papers do not establish privacy, deletion, access-control, contamination, or universal deployment guarantees.