Consolidation: From Repeated Facts to Evidence-Backed Beliefs
Hindsight's retain path creates connected raw facts, but raw facts alone do not provide a stable answer to a recurring question. This chapter follows the next lifecycle: background consolidation turns unconsolidated facts into observations that can be read as patterns while keeping their supporting evidence. The key judgment is why this is knowledge formation rather than duplicate cleanup, and how scopes, proof, replacement, and invalidation keep a convenient belief from becoming an untraceable one.
Consolidation is a second lifecycle
A retain request can finish before an observation exists. That is intentional. Retain's immediate job is to create the raw world and experience facts and the links required to retrieve them. Consolidation is a background process that later selects facts not yet considered for observation formation. It reads those facts in batches and asks a language model to decide whether to create a new observation, update an existing one, merge a near-duplicate, or delete an observation that no longer has support.
An observation is not simply a shorter copy of a fact. It is a statement about a pattern across facts. If several conversations say that a user prefers concise technical explanations, the raw layer may contain several differently worded reports. An observation can express the shared preference once, while retaining the source memory IDs that support it. If later evidence says the preference applies only during incident response, the observation can be refined rather than treated as a permanent rule.
The observation record carries more than its visible sentence. Hindsight stores the IDs of supporting source memories, a proof count, temporal bounds inherited from those sources, and history about how the observation changed. Those fields let a later reader ask not only “what does the bank believe?” but also “which memories led to this belief, how many distinct sources support it, and across what period?” The observation is a usable summary with an evidence trail behind it.
The create, update, merge, and delete choices have different meanings. A create says that the new facts support a pattern not represented by an existing observation. An update says that new evidence changes the wording or support of an existing pattern. A merge says that two observation candidates are really one conclusion and that their source sets should be folded together. A delete says that an observation no longer deserves to remain active, usually because its support disappeared or an explicit reconciliation removed it. These actions affect derived rows; they do not rewrite the raw facts that were used to make the decision.
The consolidation worker prepares its slow work without holding a database transaction open across every model call, then commits the related observation writes and consolidation markers together. That boundary matters when a model call fails or a worker stops. A failed batch should not leave an observation claiming support that was never committed, and a successfully committed observation should not lose the record that its source facts were processed. The result is not perfect availability—consolidation can be delayed or fail—but a clearer distinction between pending knowledge and committed knowledge.
The background boundary creates both an advantage and a condition. The advantage is that ordinary retention need not block on every higher-level synthesis step. A caller can begin using the raw facts while consolidation catches up.