From Conversation to Durable Fact
This chapter follows Hindsight's retain operation, the point where incoming conversation or documents become memory that later retrieval can use. The practical question is not how to save text; it is how to decide what the text means, whose experience it describes, when it happened, and which relationships should survive the original wording. Understanding that pipeline lets a reader predict both Hindsight's faster read path and the costs and errors introduced before a query is ever asked.
Retain does not save a blob—it prepares memory
A passive storage API can accept a string, assign it an identifier, and return success. Hindsight's retain path has a different contract. It treats the input as material from which significant facts must be extracted and made searchable. The system can retain a conversation, a document, or ordered text and attachment content. Before it can become a useful memory, that content is divided into manageable chunks while preserving stable identities for the document and its chunks.
Chunking is more than a convenience for fitting a prompt. Stable chunk identities let Hindsight tell which part of a document changed during a later retain. If a document is re-sent with one paragraph edited, the system can avoid treating every unchanged paragraph as new work. That matters for cost and for continuity: an unchanged piece can keep its existing facts and links instead of being needlessly reinterpreted. The chunk boundary is therefore part of the system's state, not just a temporary formatting choice.
The next stage uses a language model to extract significant facts. In its ordinary concise mode, Hindsight asks for the essence of information worth remembering over time rather than every conversational filler. The extracted representation can include what happened, when, where, who was involved, why it matters, a perspective classification, named entities, and optional causal relationships. A sentence such as “Alice joined Google last spring and was thrilled about the research opportunities” is not reduced to a bare employment fact. The extraction is intended to preserve the event, its time, and the motivation or emotional context that might matter to a later question.
Consider what this means for a support conversation. A user may report that a deployment failed after a configuration change, explain that the change was made to reduce latency, and mention that the rollback restored service. The useful memory is not simply the last sentence or the most distinctive noun. It includes the incident, the action, the reason for the action, and the outcome. Those dimensions give later retrieval several entry points: a query about the outage, the configuration, the latency goal, or the rollback can all reach related material. They also make the memory legible to a human operator who needs to understand why a later observation was formed.
Context supplied with the content helps decide whose words are being recorded and how much surrounding meaning should be retained. A transcript in which a customer says “I bought a Tesla” should not be interpreted as the agent's own experience merely because the sentence is first person. Conversely, a log in which the assistant says “I changed the retry policy” should preserve that as an agent action. The extraction prompt can use this context, but the caller remains responsible for supplying an accurate description. Hindsight can preserve a distinction; it cannot recover a speaker attribution that the input never made clear.
That interpretation is what makes later questions possible, but it is also the first important boundary. The model decides what counts as significant and how to phrase it. Hindsight provides a structured schema, prompts, validation, and sanitization; those controls constrain the output but do not make the model omniscient. A retain request can therefore succeed while producing no facts if its mission excludes everything in the input. The document may still be stored, but recall and reflect search memory units, so a document with zero extracted units is not reachable through those operations until it is reprocessed under a broader policy.
After extraction, Hindsight generates embeddings for the resulting facts. An embedding is a numerical representation used to compare meaning, not a replacement for the fact text. It gives semantic retrieval a route to related wording, while the text and metadata remain available for other retrieval strategies. Hindsight also resolves entities and prepares relationships. These steps turn the extracted units into candidates for a connected memory structure rather than isolated search results.
The write path separates expensive reads from the transaction that makes the memory visible. Entity resolution and some similarity work can be prepared before the final write. The facts and retrieval-critical links are then inserted together so that a newly visible memory does not permanently lack the relationships required to find it. The system creates temporal, semantic, and causal links as part of that preparation. Once the retain operation completes, observation consolidation may continue in the background; the caller does not need to wait for every higher-level belief before the raw facts exist.
This separation also clarifies ownership. The extraction model proposes facts and metadata. The embedding and entity stages propose numerical and relational handles. The durable store owns the committed memory units and links. The background worker owns later consolidation, not the immediate truth of whether the raw retain succeeded. Keeping those responsibilities distinct helps Hindsight retry or reprocess work without confusing a transient model failure with a missing document. It also explains why a caller can receive a completed retain before an observation exists: the raw write and the derived write have different triggers and lifecycles.
For a large document, the same design is applied repeatedly rather than by sending an unbounded prompt to one model call. Hindsight can split oversized content into sub-batches, process them asynchronously, and preserve the document identity across those slices. The result is a single logical document whose facts arrive in stages. That improves throughput and recovery, but it means operators must distinguish “the operation was accepted” from “all downstream consolidation has caught up.”
The sequence has two different success conditions. The immediate success condition is that the submitted content is processed and the resulting facts and links are durably stored. The longer-term success condition is that the facts can be retrieved and later consolidated into useful knowledge. Those conditions can diverge. A syntactically successful request can produce no facts; a fact can be present but poorly extracted; or a document can be re-ingested while its old observations need to be invalidated and rebuilt. Retain is the beginning of the learning loop, not a guarantee that the loop has already learned the right thing.
Stable document identity is what makes correction possible. When the same document is retained again, Hindsight can compare its new chunks with the stored version instead of blindly appending a second copy. Changed chunks are re-extracted; unchanged chunks can survive with their existing facts and links. If a chunk or whole document is replaced, observations that depended on the outgoing facts cannot be left standing as if their evidence were still present. The retain path therefore includes cleanup and invalidation work around replacement. This is an important difference between updating a memory source and simply adding another message to a log.
The approach preserves a useful asymmetry: new evidence can be added without destroying the historical record, while derived knowledge can be withdrawn and rebuilt when its sources change. It also creates a maintenance obligation. A caller that omits stable document identity can turn repeated ingestion into duplicate documents. A caller that changes extraction policy must understand that reprocessing may produce a different set of facts. Idempotent storage protects the system from some retry failures; it does not make two different interpretations of the same source equivalent.
Perspective, time, and policy determine what survives
A memory is more useful when Hindsight knows whose experience it represents. The extraction rules distinguish world facts from the agent's own experience. “Alice works at Google” is about the world, even if a user told the agent that information. “I recommended Python to Alice” is about what the agent did. This distinction is operational: a later agent may want to know what the user prefers, what the assistant previously attempted, or what happened independently of either one. Treating all first-person language as the agent's experience would corrupt that boundary, so the surrounding context can identify who is speaking.
Time has a similar two-sided structure. Hindsight records when an event occurred and when the system learned about it. Suppose a user tells the agent in January that Alice got married in June of the previous year. The event belongs in a historical query about June, while the January mention can still matter for recency or for understanding when the bank acquired the information. Collapsing those timestamps would force the system to choose between two wrong behaviors: making an old event impossible to find by its actual date, or treating every newly mentioned old event as if it just happened.
The extraction rules also preserve the precision of a date. A statement that says an event happened in a particular year is not silently converted into a specific day. Coarse periods can be represented as spans, while a point event can carry a narrower date. Relative expressions can be resolved against the event date supplied to the retain call rather than against an arbitrary later clock. This is the kind of detail that feels invisible when it works and becomes damaging when a historical answer depends on it.
Hindsight can steer what the extractor pays attention to with a retain mission. A bank might ask it to preserve technical decisions, API design choices, and architectural trade-offs while ignoring greetings and meeting logistics. The mission narrows extraction; it does not turn the raw document into a different source. This distinction matters because a narrow mission can trade away later retrieval of material it excluded. If the bank needs that material later, the stored document can be reprocessed without uploading it again, but reprocessing applies the current extraction configuration.
Extraction modes make a similar trade. Concise extraction favors selective, efficient facts. Verbose extraction keeps richer context. Custom instructions let a bank define its own extraction rules. Verbatim mode preserves the original chunk as the fact text while still extracting metadata, and chunk mode stores chunks without an LLM extraction call or extracted metadata. These choices answer different needs. A support history that requires exact wording may need verbatim preservation; a personal assistant that needs durable preferences may benefit from selective facts; a large corpus may prioritize cost and throughput.
None of these modes is simply “more accurate.” Verbose output can preserve useful nuance but costs more context and may create more material to rank. Concise output reduces noise but can omit a detail that later turns out to matter. Chunk-only storage avoids the extraction call but gives up the structured fields that make entity, temporal, and causal retrieval possible. The right choice depends on what the bank is meant to remember, not on a universal preference for more or less text.
A retain mission makes that choice explicit at the bank level. For a project bank, a mission focused on architectural decisions can prevent social chatter and meeting logistics from consuming extraction and retrieval capacity. For a personal assistant, the same mission would be too narrow if the agent is expected to remember preferences, commitments, or important life events. The mission is therefore part of the bank's memory policy, not merely a prompt tweak. It determines which source material can later participate in observations and reflective answers.
The zero-fact case is especially revealing. Hindsight can report that the retain completed even though its extraction produced nothing worth storing as a memory unit. That behavior is coherent if the caller asked for a tightly focused bank, but dangerous if the caller assumes completed means everything is now searchable. Operators need a way to notice the difference and reprocess the document with a broader mission when the omission was accidental. The system keeps the source available for that recovery, but it does not silently invent a fact to make the request look successful.
The same boundary appears in multilingual and multimodal content. Hindsight's extraction rules aim to keep facts in the language and script of the input rather than translating them by default. With ordered image or file blocks, the extractor can read an attachment beside the prose that refers to it, and a fact can carry the specific attachment it used as evidence. That provenance is deliberately narrow: a screenshot next to a paragraph is not automatically evidence for every fact in the paragraph. The system's goal is not to decorate a memory with all nearby media, but to preserve the connection that the extraction actually relied on.
What, then, is fixed by the software and what remains a judgment? Chunk identity, document hashing, transaction boundaries, result alignment, replacement cleanup, and background task scheduling are system controls. They make retries, updates, and visibility more predictable. The choice of which facts matter, how a speaker should be classified, whether two names refer to one entity, and which cause or date is present remains model-assisted interpretation. Hindsight can detect malformed output and apply policy, but it cannot turn an uncertain inference into a verified fact merely by storing it in a database.
This division suggests how to evaluate a retain result. First ask whether the source was preserved and whether the document identity makes a retry or reprocess safe. Then ask what fact was actually extracted: whose perspective does it represent, what date does it claim, and what context did the extraction keep? Only after those questions should a reader judge whether a later search or observation is good. A polished downstream answer cannot repair a missing source fact, an incorrect speaker, or a date that was normalized to the wrong event.
That is why the next stage of the series matters. Retain gives Hindsight durable units with perspective and metadata; it does not yet give those units the full behavior of memory. Their value increases when entities, dates, causes, attachments, and visibility rules connect them to one another. The next question is how those connections are built—and how Hindsight can use them without pretending that every inferred relationship is certain.