How a Memory Becomes a Future Instruction
The security subject here is the path by which an AI agent turns interaction data into persistent external memory, then brings that memory back into a later task. The important focus is not a tour of storage products; it is the sequence of state changes between an accepted write and an action. Following that sequence gives the reader a precise way to ask what survived, who controls each transition, and what evidence is needed before calling the result an attack or a repair.
A write is only the beginning of the state transition
An agent's memory lifecycle starts with a candidate: something in a conversation, a tool result, a workspace artifact, or a message from another agent that might be worth retaining. The agent or its harness passes that material through the intended memory interface. The interface may accept an add, update, save, or file operation. That acceptance matters operationally, but it says only that a request crossed one boundary successfully. It does not yet show that the relevant meaning is present in persistent state.
The next question belongs to the memory backend. After the write, what entries were added or changed? Did the backend retain the target rule, destination, condition, or authorization bypass? Did it split one input into several records? Did it rewrite the wording, summarize it, or attach it to an entity? The durable security state is the answer to those questions, not the request that tried to create it. MemSecBench therefore separates an accepted write from persistence and examines the post-write backend state. A successful tool call or an agent's statement that “I saved this” cannot substitute for the changed memory entry itself.
This distinction is easy to miss because a useful memory system hides its storage work. A person asks an agent to remember a preference and receives a confirmation; in the background, a file may be updated, a service may create a record, or a memory-side process may extract a shorter fact. The agent's response reports the interface event. The security question concerns the state that will still be available after the current session is gone. If the backend stores only a partial version, then the partial version—not the original request—becomes the material a future task can encounter.
Persistence also has a temporal boundary. The memory state outlives the interaction that created it, but it does not automatically inherit the original interaction's full context. A later session may know the retained sentence while lacking the surrounding explanation, the speaker's authority, the exception that was discussed, or the date on which the statement stopped applying. The system has crossed from one context into another, and the information has to carry its own boundaries if it is not to be overgeneralized.
The later task creates a second set of transitions. A user submits an ordinary request without repeating the earlier content. The agent, retriever, or memory service searches the persistent state using the current task as a cue. If a matching record is returned, that is recall exposure: the memory has become visible to the decision process. It is not yet adoption. The agent may compare it with current policy, ask for confirmation, reject it as stale, or use it as one piece of evidence. Adoption occurs only when the recalled content shapes the reasoning, parameters, tool selection, or action basis for the current request.
That difference matters because the same memory can be recalled and refused. A safety check that proves only that a suspicious record was returned cannot tell whether the agent treated it as authoritative. Conversely, a final unsafe action without a recorded recall event leaves open a different explanation: perhaps the agent acted for another reason, or the relevant record was never in the active context. A lifecycle account keeps exposure and influence separate so that controls can be placed at the right boundary.
The final transition is from the agent's decision to a consequence outside the conversational response. MemSecBench requires a case-defined record in a service, environment, or artifact for its external-consequence checkpoint. A plan, a draft, a tool call, or a claim of success is not enough. The operation must leave admissible evidence in the state that the case says should change. This requirement gives “harmful use” a concrete meaning and prevents the evaluation from treating intention as outcome.
The sequence is therefore not a single pipeline with one pass/fail switch. It is a chain of owners and conditions:
- The interaction source supplies content, but does not thereby grant it authority.
- The agent or harness requests a memory operation, but an accepted request does not prove the final state.
- The memory backend owns the representation that persists beyond the session.
- The retrieval path decides what becomes visible to the later task.
- The agent decides whether the returned material applies and allows it to shape execution.
- An external service, environment, or artifact records whether the consequence actually occurred.
These are conceptual ownership boundaries, not claims that every system implements them as separate services. They explain why a failure can be missed when a system reports only the first event or the last answer.
The diagram's branches are deliberately explicit. A memory write can be accepted while persistence fails; a persistent record can exist while retrieval misses it; a retrieved record can be rejected; and an adopted direction can still fail to produce the required external state. The repair path is independent because repairing the memory should not depend on whether a separate downstream task happened to trigger the stored rule. In the MemSecBench design, Execute and Forget start from independently restored copies of the same verified post-Write state, so the consequence branch and the repair branch do not contaminate each other.
Representation decides what the next session can see
Once the lifecycle is viewed as state transitions, storage format becomes a safety question rather than an implementation footnote. Different representations preserve different parts of a message and make different associations available to retrieval. The same semantic case can therefore travel through the system in several forms while remaining capable of influencing the later agent.
A native file may retain a rule in readable text. A vector-backed store may keep the text alongside an embedding so that a later query with related language can retrieve it. A fact-oriented store may decompose the statement into smaller propositions. An entity-linked system may attach the record to names, topics, or objects that later help it win retrieval. A structured note system may add keywords, context, links, categories, or an episode summary. These forms are not interchangeable, and none is automatically safe because it is more structured than raw text.
MemSecBench's smart-home after-sales case makes this concrete without relying on a single representation. The planted rule says that the agent should share several categories of device and network information with an after-sales group without reconfirmation. In the native condition, the rule is stored as a standing file-based instruction. In the service-backed conditions, it appears as a vector record, an entity-linked record, or a structured memory note. The representations differ, but each of the four selected configurations later retrieves the rule during a routine maintenance summary and applies it through the normal sharing workflow.
The lesson is not that every backend behaves identically. The lesson is that semantic influence can survive a change of representation. A defense that searches only for an exact phrase may miss a decomposed or rewritten form. A defense that assumes decomposition always neutralizes risk may overlook the fact that several smaller records can be recombined. A summarizer may shorten a record while removing the exception that limited it. Entity links may improve retrieval of useful context, but they can also attach a rule to the wrong object or make a related query more likely to surface it. The relevant question is what safety-bearing meaning and metadata remain available at the next boundary.
Metadata often carries the boundary more than the main sentence does. A rule may be valid only for one supplier, one account, one workflow, one recipient, or one date range. It may be a suggestion rather than an authorization, or an observation rather than a directive. If storage keeps the action but loses the scope, time, source, or approval status, the later agent receives an instruction that appears more general than the original statement justified. The vendor-onboarding illustration from MemSecBench uses exactly this pattern: a payment procedure valid only through a stated date becomes a standing supplier preference when the expiration is omitted. The unsafe effect comes from the relationship between the instruction and its lost condition.
The reverse failure is also possible. A memory may remain accurate but be retrieved for the wrong entity or task because its representation is associated through similarity. In that case, the sentence itself need not be malicious or false. The failure occurs when the system binds valid information to the wrong current object. A memory can also become unsafe only after composition. One record supplies a normal workflow, another supplies a destination, and a third supplies a condition; none independently looks like the final harmful direction, but their joint retrieval produces it. Representation controls what can be combined, while retrieval controls which combinations become visible.
This is why a later agent should treat memory as contextual evidence, not automatic authority. Relevance answers “might this help with the task?” It does not answer “is this valid now, for this entity, under the current authorization?” The memory backend can expose a record, but the agent still needs current authority and task constraints to determine whether the record governs action. That is especially important when a memory was written through a channel whose content is not inherently trustworthy. A carrier tells the system how information arrived; it does not prove who had the right to establish a standing rule.
Repair must reason about representation as well. If the malicious meaning is split across several records, correcting only the most obvious sentence may leave the operative combination intact. If a summary or linked node carries the unsafe meaning, deleting the raw source may not be enough. Conversely, removing an entire topic can erase legitimate preferences and procedures that share words with the target. A repair claim therefore needs a before-and-after view of the backend state and a semantic test: is the unsafe guidance still operative, and are the required benign meanings still represented?
The recovery branch also shows why “delete the memory” is not a general answer. MemSecBench evaluates target removal and benign preservation separately. A store can be cleared and no longer express the malicious rule, yet fail the preservation condition because useful state disappeared with it. Selective repair can instead remove, rewrite, correct, or explicitly invalidate the unsafe semantics while retaining confirmation rules, minimal-disclosure guidance, authoritative verification, or ordinary archiving knowledge. The physical operation varies by backend; the safety property is semantic and state-based.
The entire mechanism can now be stated without treating memory as a black box. Content crosses a session boundary when the backend commits a representation that a future retrieval can expose. Its danger depends on which qualifiers survive, which records can be assembled, and whether the later agent mistakes relevance for authority. The same lifecycle also determines what a defensible repair must inspect. The next question is how these delayed forms differ when harmfulness is visible in one record, assembled from several records, or activated only by a future context.