Overview

LongMem separates long-term memory into a frozen backbone, a cache memory bank, and a trainable residual SideNet. The backbone encodes previous segments into selected key-value representations and provides a stable space for current queries; the cache holds a finite rolling history; the SideNet retrieves and fuses past context with local computation before the shared frozen output head. Segment ordering preserves causality: previous context is available before current sequences are appended, and oldest sequences are evicted at capacity. Freezing the encoder addresses one source of representation staleness and limits what memory-augmented training changes, but it does not guarantee retrieval quality, complete retention, or safe deployment. The architecture’s value lies in explicit ownership of creation, residence, retrieval, fusion, and eviction.

Learning outcomes

LongMem separates encoding, storage, and reading: the frozen backbone creates representations, the cache retains them, and the SideNet learns to use them.
Its segment-level update is causal: previous representations are available before current sequences are appended, while oldest sequences are evicted.
A frozen encoder reduces one source of representation staleness, but retrieval, finite capacity, fusion, and compatibility remain failure surfaces.

Keywords

LongMemBackboneCacheSideNetRetrievalStaleness

LongMem: Turning Past Context into a Searchable Cache

LongMem takes the external-memory path seriously by separating the model that encodes the past from the components that retrieve and use it. Its central question is not how to make a frozen language model silently absorb an ever-longer input. It is how to let that model produce stable representations of earlier segments, keep those representations in a cache, and train a separate reader to bring relevant history back into the current computation. Following those ownership boundaries makes the design—and its answer to memory staleness—much easier to judge.

Three components, three owners of state

LongMem has three main parts: a frozen backbone language model, a cache memory bank, and a trainable residual SideNet. They are connected, but they do not have the same job or the same update schedule.

The backbone is the original pretrained language model. During the memory-augmented adaptation stage described by the paper, its parameters are frozen. When a previous segment passes through it, LongMem extracts attention key-value pairs from a selected decoder layer and places them in the cache. When a current segment passes through it, the backbone produces hidden states that are passed to the SideNet. In both cases, the backbone supplies a stable representation space rather than learning a new representation every time the memory-augmented objective is applied.

The cache memory bank owns the stored long-term representations. It is not a second copy of the whole language model and not a raw archive of every earlier token in its original form. It contains selected key-value material produced by the backbone. The cache can be updated as the sequence advances, but the source of the stored representation remains the frozen encoder. This separation is important because a cached vector is useful only relative to the representation space in which it was created.

The SideNet owns the learned reading process. It is a residual network attached to the backbone, with ordinary decoder layers and a special memory-augmented layer. It receives the current input’s hidden states, retrieves relevant key-value pairs from the cache, and learns how to combine the two sources. The SideNet is therefore not a replacement language model that starts over with the memory. It is an adaptation path that teaches the pretrained model how to use information that the ordinary local context cannot contain.

The paper also reuses the frozen backbone’s output head. This keeps the final language-modeling interface tied to the pretrained model while the SideNet learns how memory should change the hidden states that reach that interface. The arrangement is a form of controlled transfer: the backbone contributes learned language knowledge, the cache contributes past context, and the SideNet learns the additional computation needed to fuse them.

The most important interface is between the backbone and SideNet. LongMem adds cross-network residual connections that pass differences between selected backbone-layer hidden states into corresponding SideNet layers. In plain terms, the SideNet does not receive only its own evolving representation. It also receives a signal describing how the frozen backbone’s representation changed across layers. Adding that signal helps transfer information from the pretrained network while the SideNet learns its memory-reading behavior.

This boundary solves a practical adaptation problem.

Published by DeepMox

FAQ

What are LongMem’s three main components?

LongMem’s three main components are a frozen backbone language model, a cache memory bank, and a trainable residual SideNet. The backbone encodes, the cache retains selected representations, and the SideNet retrieves and fuses them.

How does LongMem keep cached memory causal?

LongMem keeps memory causal by allowing a current segment to retrieve only earlier cached segments and appending the current sequences after they have served as current context. Older sequences are removed as the rolling bank reaches capacity.

Why does LongMem freeze the backbone?

LongMem freezes the backbone to keep its representation space stable while the SideNet learns memory reading and fusion, limiting changes to pretrained behavior. This addresses one staleness source, not all retrieval or capacity problems.

Previous6 minTitans and LongMem: Memory Boundaries

Next6 minLongMem: Retrieval Is the Memory