Overview

LongMem’s external bank becomes usable through a retrieval chain: a current token forms a query, mean-pooled chunk keys rank contiguous chunks, and selected spans expand back into token-level key-value pairs. Separate local and memory attention results are mixed by a trainable head-wise sigmoid gate. This exposes distinct failure points: poor ranking, coarse chunks, cache eviction, incomplete representations, or a gate that suppresses useful history. Chunk size and bank size change different dimensions of the trade-off, and the reported retrieval index adds measurable cost. LongMem’s “unlimited” context claim therefore describes access beyond the local window, not lossless retention of every token. Effective memory depends on fidelity, capacity, latency, and learned fusion.

Learning outcomes

LongMem searches mean-pooled chunk keys, then expands selected chunks into token-level key-value pairs for memory attention.
A head-wise sigmoid gate decides how much retrieved memory changes each local attention result, so retrieval and fusion can fail separately.
Finite cache residence, chunk granularity, and index cost mean “unlimited” context is not lossless recall.

Keywords

LongMemRetrievalChunksAttentionGatingCapacity

Retrieval Is the Memory

LongMem’s cache becomes useful only when a current token can find the right part of it and the model can decide how much that retrieved past should matter. This chapter follows that path from a token query to contiguous chunks, back to token-level key-value pairs, and then through two attention results mixed by a learned gate. The point is practical: an external bank does not provide long-term memory by its size alone; retrieval granularity, fusion, and capacity determine what the model can actually use.

From a token query to a chunk of memory

LongMem does not treat the cache as an unstructured archive in which every token is searched independently. It groups cached key-value pairs into contiguous chunks. For each chunk, the key vectors are mean-pooled across the chunk dimension to create a compact key for indexing. A query from the current token is compared with these pooled chunk keys, and the highest-scoring chunks are selected. The selected chunks are then expanded back into their token-level key-value pairs before they enter the memory-augmented layer.

This is a deliberate compromise between retrieval precision and context integrity. Searching individual cached tokens could offer fine granularity, but it would make the retrieval index larger and could return scattered pieces that no longer preserve the local structure of the original text. Searching chunks reduces the number of index entries and returns neighboring tokens together. The model gives up some freedom to select an isolated token in exchange for a retrieval unit that carries a small amount of surrounding context.

The process for one current token has several distinct transitions. First, the current hidden state produces a query. Second, that query addresses the index of mean-pooled chunk keys using an inner-product similarity. Third, the index returns the highest-scoring chunks rather than the final token-level records. Fourth, LongMem expands those chunks so the memory-augmented layer receives the original key-value pairs within each selected span. Only then can the current token attend to the retrieved memory at token level.

The distinction between index representation and retrieved representation matters. The mean-pooled key is a search aid; it is not the complete memory content that the SideNet consumes. A chunk can be selected because its pooled key resembles the current query, while the individual keys and values inside the chunk provide the finer information used for fusion. Retrieval therefore has two possible mismatch points: the pooled key may rank the wrong chunk, or a correctly selected chunk may contain more or less detail than the current token needs.

The choice of contiguous chunks also preserves a limited form of local meaning. A demonstration example, a sentence, or a short passage often depends on neighboring tokens. Returning the entire surrounding span can help the reader use that relationship instead of seeing an isolated label or fragment. But the same choice can return distractors. Chunking does not solve relevance; it defines the unit at which relevance is judged and the amount of context that arrives after a match.

Chunk size is consequently a workload parameter.

Published by DeepMox

FAQ

How does LongMem choose which chunks to retrieve?

LongMem mean-pools the key vectors within each contiguous chunk, compares the current token’s query with those pooled keys by inner-product similarity, and selects the highest-scoring chunks.

Why does LongMem use a gate after retrieval?

LongMem uses a head-wise sigmoid gate to mix local attention with attention over retrieved memory, so each head can decide how much distant context should influence the result.

Does a larger LongMem cache guarantee better recall?

No. A larger cache retains more candidates but can increase search or storage cost and still may rank the wrong chunk; smaller or better-matched settings can work better for a task.

Previous7 minLongMem: Turning Past Context into a Searchable Cache

Next8 minTitans and LongMem: Results by Workload

LongMem: Retrieval Is the Memory

Retrieval Is the Memory

From a token query to a chunk of memory

FAQ