Why agentmemory Treats Memory as a Runtime

agentmemory is a local-first memory runtime for AI coding agents, built as a worker on top of iii-engine. This opening focuses on the problem its design is trying to solve: how an agent can carry useful decisions across sessions without loading an ever-growing transcript into every prompt. By the end, you should have a working model of the system's lifecycle and enough judgment to distinguish durable memory from a bigger context window, a static instruction file, or a vector index by itself.

The cost of a forgotten decision

A coding agent can be remarkably capable inside one session and strangely inexperienced in the next. During a single task, it may inspect a repository, choose an authentication library, discover a test constraint, change a file, and repair a failed command. When the session ends, the reasoning that connects those actions is usually trapped in the transcript. A later session sees the repository's current files, but not necessarily why they look that way.

The consequence is not just inconvenience. Re-explaining a decision consumes attention, and rediscovering it can produce a different decision. The example used by agentmemory is deliberately ordinary: one session sets up JWT authentication; a later session asks for rate limiting. A useful memory of the first session would include the middleware location, the tests that validate tokens, and the reason jose was chosen over jsonwebtoken for Edge compatibility. Without that context, the second task begins with archaeology instead of continuity.

Many coding agents already have a form of memory. A project instruction file or a generated MEMORY.md can preserve durable notes, but a file is a poor substitute for a growing record of work. It must be edited, it becomes stale, and reading it usually means placing a large, undifferentiated block into the next context. Agentmemory's own comparison describes this difference as the difference between sticky notes and a searchable database: the database can keep a larger corpus, select only top results, and serve more than one agent host through MCP or REST.

That comparison does not make a static file useless. A short instruction file is valuable for rules that should be visible every time. The question is what happens to everything else: discoveries, failed approaches, implementation choices, preferences, and the sequence of a debugging session. Those records need a different lifecycle from instructions that a human deliberately maintains.

Memory is a lifecycle, not a storage bin

The important idea in agentmemory is that persistence alone does not create useful memory. A durable system has to answer several different questions in order.

First, what crosses the boundary? Hooks observe session starts, prompts, tool calls, failures, compaction events, and session endings. The system does not need an agent to remember to call a special save function after every useful action. It can receive events in the background, associate them with a session, and preserve the evidence from which a later memory may be derived.

Second, what should be trusted and retained? Agentmemory applies a privacy filter before storing observation data and attaches an origin channel such as user, agent, tool, import, or shared. This is already more precise than treating every sentence in a transcript as an equally authoritative fact. It is still not a guarantee of secrecy: the filter recognizes known private tags and common credential patterns, while the local store remains the operator's responsibility to protect.

Third, what representation can be found later? A raw tool payload is useful evidence but a difficult search document. Agentmemory creates a structured compressed observation with fields such as a title, facts, narrative, concepts, files, importance, and provenance. In its default mode this compression is synthetic and does not call an LLM. A provider-backed compression path exists, but it is explicitly opt-in because sending every observation to a model changes both cost and failure behavior.

Fourth, how much should return to the next agent turn? Retrieval is not the same as replaying history. Agentmemory can combine lexical BM25 search, vector similarity when embeddings are enabled, and graph matches when structural data exists. It then returns a bounded set, with compact forms and optional expansion, rather than assuming that the whole corpus belongs in the prompt. Its documented default token budget is 2,000, and context injection into the agent conversation is off by default. Capture can therefore be active even when automatic insertion is not.

Finally, how does knowledge change? A deliberate memory can supersede an older version, while the older record remains available for history rather than competing in current search. Session summaries, semantic facts, procedures, lessons, and insights can be derived later. Access and time can strengthen or weaken retention, and explicit forgetting or automatic cleanup can remove records and their index entries. Governance matters because a memory that cannot be corrected, traced, or deleted is not dependable simply because it is persistent.

This is why “memory runtime” is a more useful description than “vector database.” A vector index answers one retrieval question. The runtime has to coordinate event capture, state, representations, ranking, background work, and the interfaces through which an agent or operator acts on that state.

There is also a difference between an observation and a memory. An observation says that something happened: a tool ran, a file changed, a prompt was submitted, or a command failed. An explicit memory says that a fact, preference, pattern, or decision is worth carrying forward. The first should preserve enough evidence to support the second, but the two should not be treated as equally authoritative. If an agent changes its decision about a library, the newer memory should become the current searchable account while the older account remains useful as history. That requires versioning and index maintenance, not just another text field.

The same distinction separates capture from injection. Capturing an event creates a durable possibility for later recall. Injecting context puts selected material into the model's next input. Combining those actions automatically may feel convenient, but it also means that every noisy or weak record can consume attention repeatedly. Agentmemory keeps the boundary visible: its capture hooks can run while context injection remains disabled, allowing an operator to inspect the corpus before making it part of every conversation.

The relationship with iii-engine makes these stages one operational system. A function can handle a memory operation, a trigger can expose it to an HTTP request or lifecycle event, state can retain the result, and a stream can make the change visible to the viewer. Scheduled work such as consolidation or forgetting can use the same runtime rather than depending on a second scheduler. That composition is the source of the system's coherence, but it is also why the engine version and its state behavior matter. A memory feature is only as dependable as the runtime that carries its transitions.

This framing changes how a failure is diagnosed. If a later session cannot recall a decision, the cause may be that the host never emitted the hook, that the payload was filtered or truncated, that the record was stored but not indexed, that an optional vector or graph path was unavailable, or that the right result was trimmed by scope or budget. A single “memory” switch cannot distinguish those cases. A lifecycle can: each boundary becomes a place to observe, verify, and improve.

flowchart LR A["Coding-agent session"] --> B["Hook or REST boundary"] subgraph I["iii-engine runtime"] B --> C["Privacy + provenance"] C --> D["Raw observation"] D --> E["Compressed observation"] E --> L["BM25"] E --> V["Vector search when enabled"] E --> G["Graph search when data exists"] L --> R["Rank fusion + token budget"] V --> R G --> R E --> M["Memory evolution"] M --> F["Currentness, decay, forgetting"] F --> L end R --> N["Next-session context"] N --> A2["Later agent session"]

The diagram also explains the role of iii-engine. Agentmemory registers functions and triggers with the engine, uses its state mechanism for persistent records, and uses streams for live updates. The repository's default configuration uses a file-backed state adapter and does not require a separately operated Postgres, Redis, or vector database. That does not mean the runtime is infrastructure-free: an engine process, writable data directory, ports, and compatible versions still matter. “Local-first” describes where the data and services live, not the absence of operational responsibilities.

Defaults are policy choices

Agentmemory's defaults reveal what its designers consider the safest starting point. With no LLM key, it runs in a no-op provider mode and still captures observations and searches them with BM25. This makes the basic continuity path predictable and keeps a provider outage from turning every tool call into a failed memory operation. Semantic embeddings are a separate choice. The project documents a local all-MiniLM-L6-v2 option that runs on the device after its first model download, as well as remote providers. Either can improve paraphrase-level recall, but neither should be confused with a prerequisite for basic operation.

LLM-written observation compression is another explicit choice. A configured provider makes model-backed operations available, but the per-observation path also requires AGENTMEMORY_AUTO_COMPRESS=true. That separation matters because background compression runs at the frequency of tool use. A model can produce richer summaries, yet it also adds recurring token cost, latency, provider dependence, and another place where structured output can fail. The default synthetic path gives up some interpretation in exchange for a simpler and more resilient capture path.

The same restraint appears in automatic context injection. Turning it on can make a new session feel more informed, but injecting memory before every tool turn can increase model input and make irrelevant context a performance problem. Keeping it off by default makes capture and injection independently controllable. An operator can first inspect what is being stored and how it is retrieved, then decide whether automatic insertion improves the workflow.

The project reports strong retrieval results under particular methods, including 95.2% R@5 on its retrieval-only LongMemEval-S evaluation with BM25 plus local vectors, compared with 86.2% for BM25-only. Those numbers are useful evidence for the value of combining signals, not a promise that an agent will answer 95.2% of questions correctly. The report explicitly distinguishes retrieval recall from the official end-to-end question-answering task. It also means that the result depends on the corpus, the metric, the version, and the index configuration.

That distinction is the boundary of the whole subject. Agentmemory can make past evidence more available; it cannot make every captured statement true, every summary complete, or every retrieved item relevant to the agent's immediate decision. Its most defensible contribution is governed continuity: a way to carry work forward while keeping capture, selection, evolution, and correction as separate responsibilities.

The next question is therefore concrete rather than philosophical: when a live agent uses a tool, what exactly does agentmemory capture, what does it filter, and how does that event become a searchable observation without blocking the work that produced it?

References