From Tool Turn to Searchable Observation

This chapter follows agentmemory at its most important boundary: the moment a live agent event becomes durable state. The focus is not a catalog of hooks, but the causal path from a host's JSON event to a sanitized, attributed, searchable observation. Understanding that path lets you judge what automatic capture really promises, why it is designed not to block coding work, and when richer LLM compression is worth the added cost and failure surface.

A hook captures an event, not a conclusion

A coding agent produces more than final answers. It opens files, edits them, runs commands, receives errors, submits prompts, and sometimes compresses or ends a session. If memory waits for the agent to summarize each important moment, capture becomes a second task that is easy to forget. agentmemory instead places its first boundary at the host's lifecycle hooks.

The bundled integration treats a session start, a prompt submission, tool activity, failures, compaction, subagent activity, and session completion as different kinds of evidence. The distinction matters because an event carries a different meaning depending on where it occurs. A user prompt records intent. A post-tool result records an action and its outcome. A tool failure records a path that did not work. A session end signals that the system can now attempt a higher-level account of what happened. The runtime can preserve these signals without pretending that any one of them is already a durable fact.

Consider a tool call that edits an authentication middleware file and then runs a test. The post-tool hook receives the host's event as JSON, extracts the session identity, tool name, input, and result, identifies the current project and working directory, and sends a normalized request to the local REST service. The hook does not need to understand whether the edit was wise. Its job is to preserve enough context for the server-side pipeline to classify and compress later.

The REST boundary performs a different job. It checks that the event includes a hook type, session ID, project, working directory, and timestamp before routing the payload to the memory worker. This separates an unreliable external caller from the internal state model. A malformed event is rejected rather than being allowed to create an ambiguous record. A valid event reaches mem::observe, which assigns an observation ID and associates the event with the session.

Session registration gives the observation a home. At session start, the server creates or updates a session record containing the project, working directory, start time, status, and observation count. Subsequent observations update that count. The system also has a recovery path for integrations that send an observation before creating a session: when the payload includes the project and working directory, the observation path can create the session implicitly. That is a small but important systems decision. It keeps an imperfect host adapter from producing invisible records that cannot later appear in session lists or summaries.

The hook is also deliberately kept out of the agent's critical path. The post-tool script dispatches its HTTP request with a short timeout, does not wait for the response to finish the coding turn, and exits after giving the local request a brief chance to flush. If the memory server is unavailable, the hook swallows the error. The agent can continue editing even when the observer is down. This is good availability behavior, but it creates a clear trade-off: automatic capture is best effort. A memory runtime cannot recover an event that the host never emitted or that disappeared while the service was unreachable.

The same separation appears at the end of a session. The client hook posts a session-end request, while the server marks the session completed and emits a session-stopped event. Downstream work—such as summarization, graph extraction, or consolidation—belongs to that server-side lifecycle rather than to a collection of subtly different client scripts. The event becomes a trigger for later processing, not a demand that the agent wait for every derivative record before it can stop.

Pre-tool activity illustrates why “supported hook” and “captured searchable fact” should not be treated as synonyms. Agentmemory includes a pre-tool enrichment path, but its current behavior is off by default because writing recalled context to every tool turn can consume model input tokens. With explicit context injection enabled, the hook can ask for file-related context before a tool runs. Without it, post-tool capture still records what happened in the background. The system therefore distinguishes observing a turn from changing the prompt that drives the next turn.

sequenceDiagram participant H as Host hook participant A as REST endpoint participant W as iii worker participant F as Privacy filter participant S as State store participant C as Compression participant P as Optional provider participant I as Search index participant V as Viewer stream H->>A: Send lifecycle event A->>W: Validate and route event W->>F: Sanitize and label event F-->>W: Filtered event and provenance W->>S: Store raw observation W->>C: Build synthetic or prepare LLM compression C->>P: Call provider only when enabled P-->>C: Structured result or fallback C-->>W: Return compressed observation W->>S: Store compressed observation W->>I: Update BM25 and optional vectors W->>V: Publish live event A-->>H: Return quickly and swallow timeout

The sequence shows two kinds of success. The host-facing success is that the coding action is not held hostage by memory. The memory-facing success is that a valid event reaches durable state and later indexing. These successes can diverge. A tool call may complete even though capture failed; a raw event may be stored even though optional compression failed; a compressed record may exist even though a vector provider was unavailable. Those are not contradictions. They are separate transitions with separate recovery behavior.

Sanitization happens before the record becomes durable

The first internal transformation is not semantic summarization. It is sanitization. agentmemory serializes the event payload and runs it through a privacy function before writing the raw observation. Explicit <private>...</private> regions are replaced, as are strings matching a set of common credential patterns: API-key assignments, bearer tokens, recognizable provider prefixes, GitHub and AWS credentials, JWT-like values, and several package or source-control token forms.

This ordering is consequential. Filtering after indexing would still leave the original secret in the stored event or an index built from it. Filtering before the write means the primary observation record and the later compression path receive the sanitized form. The filter is not an encryption layer and it is not a promise that every sensitive value will be recognized. A novel token format, a secret split across fields, or sensitive personal information that resembles ordinary prose may pass through. Local-first storage reduces the number of places data travels, but the operator still has to secure the machine, the data directory, and any network exposure.

Sanitization also changes what “faithful capture” means. The purpose is not to preserve every byte at any cost. It is to preserve useful work context while reducing the chance that a memory system becomes a second secret repository. That necessarily introduces the possibility of false positives: a value that resembles a credential can be replaced even if it was harmless. The resulting record should be read as a privacy-filtered account of an event, not as an untouched forensic copy.

After filtering, the observation receives an origin channel. Prompt submissions are attributed to the user; tool-related events are attributed to the tool boundary; other lifecycle events are attributed to the agent. Imports and deliberate team sharing use their own channels. The origin includes a capture timestamp and, when available, the tool name as detail. This label is not a truth score. It answers a narrower question: how did this content cross into the system?

That distinction becomes valuable when a later memory is questioned. A remembered preference derived from a user prompt has a different provenance path from a fact extracted from a tool output or imported from an old transcript. Agentmemory carries origin fields into compressed observations and explicit memories, so a caller can inspect the route instead of treating every derived sentence as if it came from the same authority. Provenance does not decide whether a claim is correct, but it makes correction and review more informed.

The stored shape also records the event's modality. Textual tool input and output are the common case. If a hook payload contains a data URL, base64 image, or image path, the observation code can extract it and classify the record as an image or mixed observation. Inline image bytes are moved into a managed image store, with reference counting and disk accounting, rather than being repeatedly copied through every derived record. Optional vision embedding can run when enabled, and an LLM provider may describe the image before text compression. If the vision call fails, the system falls back to text-only compression rather than making an image failure erase the surrounding tool event.

The managed-path check is an additional boundary for images. During provider-backed description, the compressor will read a stored image only when the reference points inside the managed image store. That prevents an arbitrary path in an observation from becoming an instruction to read unrelated files. The feature remains conditional: image descriptions require a provider that supports them, image embeddings require an explicit setting, and the published text retrieval evaluations do not establish image-recognition quality.

Once the sanitized raw event is written, agentmemory creates a searchable representation. In the default path, it builds a synthetic compressed observation locally. The record has structured fields such as a title, facts, narrative, concepts, files, an observation type, and an importance value. This is not a model's interpretation of the entire project. It is a compact representation derived from the event so that later lexical search can work without an external call.

The runtime publishes the raw and compressed stages to its live stream as well as retaining the resulting observation state. That makes the transformation visible to the viewer: an operator can see that a tool event arrived and then see the representation that was indexed. It also makes the stages diagnostically distinct. If the raw stream appears but no compressed event follows, the likely problem is downstream processing rather than host capture. If the compressed event appears but a query cannot find it, the issue belongs to indexing, retrieval, scope, or budget.

Compression is a cost and reliability decision

The richer path is enabled only when AGENTMEMORY_AUTO_COMPRESS=true and a provider is available. A provider by itself does not cause every post-tool event to be sent to an LLM. This separation is easy to miss but central to the design. “The runtime can call a model” and “the runtime calls a model on every tool use” have very different cost profiles.

When the richer path runs, agentmemory can first ask a vision-capable provider for an image description when the observation contains an image. It then builds a compression request from the hook type, tool information, sanitized input and output, user prompt, timestamp, and any image description. The provider is asked for a structured result rather than an unconstrained paragraph. The response is parsed into the allowed observation fields, validated against a schema, and assigned a quality score. Modality, image references, agent identity, and origin are carried forward rather than lost in the summary.

The validation step controls shape, not truth. A response can be well-formed and still omit an important detail or misinterpret an ambiguous tool result. The quality score is a signal for the runtime's metrics and later judgment; it is not an independent fact checker. If parsing, validation, or the provider call fails, the function returns a failure result and logs the problem. Because raw capture occurred before this optional operation, provider failure does not have to become capture failure. The searchable representation may be less rich or delayed, but the event has not simply vanished.

Synthetic compression gives up interpretation in exchange for predictable local behavior. It does not require an API key, does not add a network round trip for every tool call, and remains available when a provider is rate-limited or misconfigured. LLM compression can produce more useful narratives and classifications, especially when the important relationship is implicit rather than written in a tool name. It also introduces recurring token spend, provider latency, structured-output failures, and dependence on a model's choices. Neither path is universally better; they optimize different parts of the lifecycle.

The same principle governs context injection. A captured observation can be searched later without being written into the agent's current conversation. Session-start injection is off by default, and the pre-tool enrichment script exits immediately unless explicitly enabled. This default avoids turning every tool call into a hidden prompt expansion. An operator who wants automatic context can enable it after inspecting the corpus and measuring its usefulness, rather than accepting prompt growth as the price of capture.

The practical result is a layered contract:

  • The host is responsible for emitting a recognizable lifecycle event.
  • The REST and iii boundary is responsible for validating and routing it.
  • The observation function is responsible for sanitization, provenance, session association, and durable capture.
  • Compression is responsible for producing a compact searchable form, with a local synthetic path and an optional provider path.
  • Indexing is responsible for making that form retrievable; vector indexing is conditional and can fail without invalidating the saved event.
  • Injection is a separate consumer choice, not proof that capture succeeded.

This contract also tells an operator where to look when the system behaves unexpectedly. If a session is absent, inspect host wiring and session registration. If an event exists but contains redactions, that is the privacy boundary doing its work. If it has raw provenance but no rich narrative, the compression path may be disabled or unavailable. If the narrative is present but a semantic query fails, embeddings may not be configured or populated. If automatic context is absent, that may be the intended default rather than a broken capture pipeline.

A memory runtime becomes trustworthy when these transitions remain visible and bounded. Agentmemory does not try to make every tool call wait for a perfect summary. It first protects and records the event, then offers richer interpretation as an optional stage. The next problem follows naturally: once many observations have been shaped and indexed, how can the runtime select the few that matter when exact words, paraphrases, and relationships point in different directions?

References