Three Signals, One Bounded Recall
agentmemory does not treat retrieval as a single semantic-search switch. This chapter explains how its BM25, vector, and graph paths recover different kinds of evidence, and how the runtime turns their candidates into a small response that can fit a working agent context. That distinction gives you a practical way to predict which mode will help a query, what must be configured before it can help, and why a high ranking score is not the same as certainty.
Search starts with lexical evidence
The first signal is BM25, a traditional relevance method built around words and their distribution across documents. In agentmemory, the indexed document is not just the title of an observation. Its searchable material includes the title, subtitle, narrative, facts, concepts, file names, and observation type. That choice matters for coding work because a file path or library name can be more discriminating than a broad natural-language summary.
BM25 rewards a term that appears in a candidate while accounting for how common that term is across the corpus and how long the candidate is. A query containing auth, middleware, and a file name can therefore favor an observation that actually discusses the relevant implementation rather than one that mentions “security” in passing. The index also normalizes common word forms through stemming, so a query and a record need not use exactly the same grammatical ending.
Agentmemory adds synonym expansion and prefix matching to that lexical base. Synonyms give a query a weaker path to related terms; the original term retains more weight than an added synonym. Prefix matching helps when a query uses a shortened form or when a repository contains a family of related names. The system also includes language-aware token handling: Greek, Cyrillic, Hebrew, Arabic, and accented Latin are tokenized directly, while optional segmenters improve word-level handling for Chinese, Japanese, and Korean. Without those segmenters, the system falls back to whole-run tokenization and reports a hint rather than silently claiming perfect multilingual recall.
This is why BM25 is more than a fallback of last resort. It is particularly strong when the question contains identifiers, exact file names, error messages, command names, or terms that appeared in the original work. A query such as “JWT token validation middleware” can benefit from literal overlap. It is also predictable: keyless agentmemory installations do not need an embedding key or a model download to make observations searchable. A local service can capture a session and provide keyword recall even when no cloud provider is configured.
The limitation is equally important. Lexical relevance does not understand every paraphrase. If a past observation says that a database query suffered from repeated per-record fetches and the later question asks about “performance optimization,” the most useful words may not overlap. Stemming and synonyms can widen the lexical bridge, but they cannot guarantee that two differently phrased ideas meet. BM25 also cannot by itself decide whether a sentence is a current decision, a rejected alternative, or an explanation of an old bug. It finds textual evidence; later lifecycle and provenance mechanisms help interpret its status.
The lexical index is therefore the stable floor of the retrieval system. It makes the basic path work with no provider and remains valuable when richer signals are unavailable. It also gives the other streams a concrete candidate source and a useful independent signal against which semantic matches can be checked.
Vectors and graphs fill different gaps
The second signal is vector similarity. An embedding provider maps an observation and a query into numeric representations, and the vector index ranks candidates by cosine similarity. The intended benefit is paraphrase: two records can be close because they express a related idea even when they do not share many words. A query about optimizing database access may retrieve a session described in terms of an N+1 query fix, provided the embedding model has captured that relationship.
That conditional phrase does real work. Vector retrieval exists only when an embedding provider is configured and the index has been populated. Agentmemory documents a local all-MiniLM-L6-v2 option that runs on the device after its first model download, as well as remote providers. A fresh keyless installation has no vector stream, even though the code knows how to use one. Enabling the local option also introduces first-use network access and local inference work. A provider that is configured but failing, returning the wrong vector dimension, or not yet indexed does not give the same behavior as a healthy, populated vector index.
The system protects the primary save path from this dependency. Embedding failures are logged and skipped rather than making a memory save fail. That makes the vector stream an enhancement to recall, not the only route to persistence. It also means that a corpus can be mixed during recovery: some records may have lexical coverage but no vector, while a later rebuild gradually fills the semantic index. Operators should read “semantic search enabled” as a runtime state to verify, not as a permanent property of the package.
Vectors are useful for meaning, but they are not explanations. A cosine score says that two representations are close under a model; it does not identify the file, decision, or causal relationship that made them relevant. For that, agentmemory can add a third signal: graph retrieval.
The graph path starts with entities and relationships associated with observations. A structural heuristic can connect concepts and files without an LLM. The richer entity-and-relation extraction path is conditional on graph extraction being enabled and a non-no-op provider being available. This distinction prevents a simple link such as “concept appears with file” from being presented as if a model had established a causal relation such as “library choice was made for Edge compatibility.” Both can be useful, but they carry different evidence strength.
When graph data exists, a query can contribute named entities and retrieve observations attached to matching nodes. The retrieval code then follows weighted paths through related nodes, preferring stronger relationships and shorter paths. The result can include graph context, so a caller sees not only that an observation matched but also the path that connected it to an entity. A query about a particular file, library, or project can therefore recover a nearby decision even when the observation's prose uses different words.
Graph retrieval also has a temporal dimension in the underlying model. Edges can carry commit or validity times and can be separated into current and historical relationships. That makes “what was true then?” a different question from “what is connected now?” The existence of fields for temporal state does not mean every observation has a complete timeline; missing graph data, stale edges, and incomplete extraction remain possible.
The graph adds structure at a price. It depends on entities being detected, names being matched, and relationships being persisted correctly. LLM-derived relations inherit model uncertainty. Structural relations can be too broad. Graph traversal can also be more operationally demanding than a bounded lexical lookup. The implementation includes pagination, snapshots, targeted indexes, and warnings for large or legacy corpora because full graph enumeration can exceed the underlying engine's response or heartbeat budgets. A graph result is a helpful route to evidence, not an assurance that the whole graph is always available at query time.
A useful mental test is to assign each signal a different question. BM25 asks, “Which records use the words and identifiers that matter here?” Vectors ask, “Which records express a similar idea?” Graph retrieval asks, “Which records are connected through the entities and relationships involved?” Agentmemory's design is strongest when those questions complement one another rather than when one stream is expected to solve every query.
Fusion is a compromise, not a vote on truth
The streams do not produce directly comparable scores. BM25 scores reflect term statistics, vector scores reflect geometric similarity, and graph scores reflect path and edge properties. Agentmemory therefore combines ranks through weighted Reciprocal Rank Fusion, or RRF. With the documented constant k=60, a candidate receives more credit for appearing near the top of a stream, and configured weights determine how much each active stream contributes.
The fusion is normalized against the streams that actually returned results. If no vector provider is active, the absent vector stream does not silently penalize BM25 candidates. If graph data is missing, the system can still return a lexical or vector result. When one observation appears in more than one stream, a small agreement bonus rewards that convergence. This is a sensible engineering choice, but it should not be mistaken for independent confirmation of truth. The streams may share the same underlying observation, and a model may produce both a vector representation and a graph relation from the same imperfect input.
Suppose a later agent asks why a particular authentication library was selected. BM25 may find the file path and the library name exactly. A vector stream may rank a narrative that calls the issue “runtime compatibility” rather than repeating the phrase “Edge support.” A graph path may connect the library, the middleware, and the decision observation. Fusion can put the record that satisfies several of those routes near the top, while preserving a way to inspect the contributing context. If only the file name matches, the record can still surface through BM25. If only the paraphrase is visible, the vector path can rescue it. If the query names an entity connected to a related observation, the graph path can widen the search.
The result is then diversified by session. Agentmemory limits the normal selection to at most three results from one session before filling remaining slots from other sessions. This prevents a verbose debugging session from crowding every other piece of history out of the answer. The trade-off is intentional: a reader seeking a detailed reconstruction of one session may prefer more than three records, while a reader trying to understand a decision across several sessions benefits from breadth.
An optional reranker can refine a small head window, but it does not turn the entire corpus into an LLM judgment. The default path remains a ranked retrieval operation. Scores are useful for ordering and diagnostics, not calibrated probabilities that a candidate is correct. A top result can be relevant but incomplete; a lower result can contain the missing qualification.
Filtering happens after candidate generation because the indexes do not carry every field needed for every scope decision. The search path can filter by project, working directory, and agent identity after loading the associated observation or memory. It over-fetches candidates first so that a relevant same-project or same-agent record is not lost merely because an out-of-scope record ranked higher. That protects recall under filtering, but it adds lookups and still cannot recover a record that never entered the fetched window.
In isolated agent scope, the identity check is part of this post-retrieval boundary. A worker can refuse a read when isolation is requested but no agent identity is available, rather than treating the absence of an identity as permission to see everything. A wildcard can be requested explicitly. The broader lesson is that relevance and authorization are separate decisions: a record may be highly relevant and still not be eligible for this caller.
Bounded delivery is part of retrieval
Finding a candidate is only half the problem. The agent still has to use the result without losing the task under a flood of context. Agentmemory exposes full, compact, and narrative forms. A compact result can identify the observation, session, title, type, score, and timestamp. A narrative result adds the relevant narrative as readable text. A full result keeps the structured observation for callers that need more detail.
memory_smart_search makes this separation explicit. It normally returns compact search results and short lesson previews. A caller can then expand selected observation IDs instead of receiving every narrative at once. This resembles how a person investigates a codebase: first identify the likely files or sessions, then open the few records that can answer the question. It also reduces the chance that a high-volume session dominates the model's input simply because it generated many observations.
Token budgets add a second bound. The search path can trim returned records to a requested budget, and the context path assembles blocks from project profiles, lessons, summaries, pinned material, and important observations. It orders those blocks by recency and keeps only those that fit. The default context budget is documented as 2,000 tokens. The estimate is deliberately approximate rather than tokenizer-exact, so a budget should be treated as a ceiling heuristic, not a billing statement.
The budget changes the meaning of “recall.” A record may be present in the search corpus, ranked by the hybrid system, and still not appear in automatically injected context because an earlier block consumed the available space. That is not necessarily a retrieval failure. It is a delivery decision. The compact-then-expand path gives the agent a way to recover more detail when the first response is insufficient, while automatic injection remains off by default so operators can choose whether memory should alter every conversation.
This bounded approach also clarifies why the system does not promise to remember everything equally. It keeps a broad searchable store, but it returns a narrow working set. Lexical matches protect exact implementation details. Vectors widen the candidate set for paraphrases when enabled. Graphs can connect entities across observations. Fusion and diversification control ranking. Filters enforce scope. Budgets control what enters the context. Each step removes some possibilities in order to make the result usable.
The cost is that every bound can hide a useful record. A small limit can truncate a long causal chain. Session diversification can underrepresent one session. A project or agent filter can exclude a record whose metadata was missing or misassigned. A graph snapshot can be an approximation for a large corpus. An approximate token estimate can stop before the model's true boundary. These are not reasons to abandon bounded retrieval; they are reasons to expose expansion, provenance, diagnostics, and deliberate configuration as part of the user experience.
The most careful interpretation of agentmemory's retrieval design is therefore neither “keyword search is obsolete” nor “hybrid search understands everything.” It is a layered recovery strategy. The keyless lexical floor keeps the service useful under minimal assumptions. Optional vectors add semantic reach. Optional graphs add relational context. Rank fusion makes the streams cooperate, and progressive disclosure keeps their output small enough to act on. The next step is to put this entire chain under pressure: follow one observation from capture to recall, compare default and optional paths, and diagnose why a result can be missing even when the underlying work happened.