The Practical Contract of tgrep

The opening question was why microsoft/tgrep does work before anyone types a query. The answer is now complete enough to state precisely: tgrep builds reusable evidence about which files could match, but it protects the answer with exact matching and a protocol for keeping that evidence current. This final synthesis turns that model into a practical judgment—when to build or serve tgrep, when to inspect its coverage state, and when a direct scan is the more faithful choice.

A trustworthy answer is a chain, not a single index

Tgrep’s central mechanism is easy to summarize and easy to oversimplify. A repository walk defines the corpus. Trigram extraction records compact necessary fragments. A query plan turns a regular expression into intersections, unions, or a deliberate MatchAll. The reader looks up candidate file IDs. The full matcher reads those candidates and decides the result. A server adds a live overlay and a publication lifecycle so that the durable snapshot can survive ordinary edits.

Each stage owns a different decision. The walker decides which paths belong to the searchable or listable corpus under ignore rules, hidden-file policy, binary handling, and size limits. The builder records content evidence for the text files that the walk admits. The query planner decides whether a pattern has a safe narrowing condition, not whether the pattern matches. The reader owns the relationship between a trigram and its postings, while the path table translates an integer ID back into a file. The matcher owns regex semantics and output behavior. The server owns the transition from one published generation to the next.

That division explains why tgrep can be fast without turning its index into a second, incomplete regex engine. A distinctive literal becomes an AND of necessary fragments, so most files can be rejected before their contents are read. An alternative becomes an OR, so a later branch cannot be lost. An optional or unconstrained expression becomes MatchAll, so “no useful index clue” does not become “no result.” A PCRE-style expression may be widened for candidate selection only when the widening direction is safe; otherwise tgrep scans broadly. In every case, the full matcher remains the authority.

The same chain continues after candidate selection. The server can resolve decoded content from its bounded cache or read it from disk, then match in parallel. It can return whole lines, context, spans, binary markers, and statistics through JSON-RPC. The client translates paths from the index root back to the user’s requested scope and renders the result. A query’s visible answer therefore depends on more than the posting intersection. It depends on the corpus definition, the reader generation, the content bytes, and the output mode all agreeing about what the request means.

The live server makes the agreement temporal as well as structural. The on-disk reader describes the last published snapshot. If src/parser.rs changes, the live overlay replaces the reader’s authority for that path. If the file is deleted, a tombstone suppresses the old reader entry. If a file is new, its overlay postings add a path that the snapshot did not contain. Once the changes are flushed, the overlay entries already represented in the new reader can be pruned. Changes that arrive after the flush snapshot remain in memory because they belong to a newer state.

A reader snapshot protects another boundary that is easy to miss. File IDs are meaningful only alongside the path table that assigned them. During publication, a query may still hold the old reader while a new one is being installed. Resolving its IDs against the new reader could turn valid candidates into missing paths or different paths. Tgrep pairs candidate IDs with the same reader snapshot that produced them. That is not a promise that the filesystem is frozen; it is a promise that one query does not mix generations internally.