From a repository to a usable graph
Ripwire’s first engineering problem is not ranking. It is deciding what a repository is allowed to contribute to a ranking, then turning that contribution into stable facts. This chapter follows the project from its crawl boundary through tree-sitter extraction to the symbol and reference records that feed the graph. The practical result is a way to tell the difference between “ripwire found nothing,” “ripwire could not see this,” and “the source itself did not provide enough evidence.”
That distinction is the foundation of trustworthy context compression. A compact answer is only useful if the reader knows what was compressed and what was never present in the first place. The map is not an abstract picture of an entire repository; it is a reproducible observation of a deliberately defined corpus.
The corpus begins with a boundary
A repository contains more than the source that an agent should read. It may include build products, vendored grammars, package caches, generated bundles, binary assets, test fixtures, benchmark dumps, and files written by the analysis tool itself. Feeding all of that into one index would not create a neutral view. Generated output can repeat the vocabulary of every question. A vendored parser can contribute huge numbers of symbols without being part of the application being changed. A binary asset cannot be made more informative by assigning it an “unsupported language” label.
Ripwire therefore begins with a crawl policy. It collects candidate paths, applies its committed directory rules and file tests, and consults the repository’s own ignore verdict when running in a Git work tree. Only after that filtering does it sort the surviving paths in byte order. The ordering is not cosmetic. File positions become the stable space from which symbol identifiers are assigned; if directory traversal order or thread timing could choose those identifiers, the same repository could produce different graph nodes on different runs.
The policy has several kinds of boundary, and they mean different things. A built-in denylist prunes ordinary noise such as build and dependency directories. A user-supplied exclusion can remove a path deliberately. A repository’s .gitignore can hide a file that the repository itself has declared uninteresting. An unsupported source-like extension means the build has no grammar for that kind of text. A non-text extension—an image or archive, for example—is treated as an asset, not as a failed parse. Oversize files are counted separately because the cutoff is a resource and safety decision, not evidence that the file has no content worth reading.
The default generic file ceiling is 4 MB. The project gives JSON a separate 256 KB ceiling because large JSON files wearing a configuration extension are commonly data sets or exports, and indexing their keys can overwhelm both the symbol table and the parse budget. YAML gets a 512 KB ceiling: it has the same data-versus-configuration problem, but a lower JSON-sized limit would discard real hand-maintained workflows. The point of these values is not that 4 MB, 256 KB, or 512 KB is a universal definition of useful source. It is that a threshold should sit at a measured pathological tail and should be named when it removes something.
The --skipped view exists for this reason. It distinguishes an oversize file from an excluded file, an unsupported extension from a pruned subtree, and an indexed file with parser errors from a file that was never parsed. A subtree that the walk stopped before entering is not reported as empty; its contents are unknown. An indexed file with tree-sitter error nodes is not automatically declared invalid source either. Parser recovery is a fact about what the parser saw, not a verdict about whether a language construct is legal in the project’s dialect. This vocabulary prevents a short map from being misread as a complete map.
The accounting is as important as the labels. For files the crawler enumerates, the indexed, oversize, excluded, and ignored populations are kept distinct. For directories pruned before enumeration, the tool reports the subtree itself because it cannot honestly count the files inside. For unsupported extensions, it can summarize the extension without implying that a missing parser is the same as a missing file. That makes a result diagnosable: if a task concerns a generated bundle that the denylist removed, the right response is not to trust the absence of a symbol. It is to change the crawl scope or inspect the artifact directly. If the task concerns a language with no grammar, the graph is incomplete before ranking begins. If the file is present but degraded, the agent can decide whether the error region touches the requested symbol. Each case changes the next action, which is why collapsing them into one “not found” state would save a few bytes while losing the more valuable fact.
Safety is part of the same boundary. Directory symlinks are not followed, so a link cannot create a recursive walk. File symlinks are checked after canonicalizing both the crawl root and the target, with a component boundary rather than a raw string prefix. That protects against a tracked link whose textual path appears inside the repository while its target lies outside it. The rule is deliberately narrow: hard links and mounts are indistinguishable from ordinary files to a path-based check, so the project does not claim that one canonical-root test solves every filesystem concern.
The parser boundary is protected in a similar way. YAML and Markdown use external tree-sitter scanners with finite serialization buffers. The repository records cases where hostile nesting could abort the process or corrupt memory in a Release build if the scanner’s own assertion disappeared. Ripwire prescans those shapes and refuses them before parsing, while also carrying targeted vendor patches with drift checks. Kotlin receives a comparable pre-parse guard for deeply nested string templates. A file that trips one of these protections appears as a disclosed refusal rather than taking down the index of every other file.
This is the first system-level lesson: coverage and safety are not opposing features. A tool that silently reads too much can be both slower and less trustworthy. A tool that skips something defensively can remain useful if it says what was skipped, why, and whether the omission affects the question at hand.
The diagram shows why the repository boundary belongs before the graph. The graph cannot rank a file that the crawl excluded, and no later disclosure can recover a file that was never admitted. Conversely, a file can be admitted but carry a health warning; that warning travels with the facts rather than being converted into a misleading zero.
Symbols are the durable units
Once a path survives the boundary, ripwire does not treat the file as one indivisible document. It parses the file and extracts a small set of structured records. A definition says that a symbol exists, what kind of symbol it is, where it sits, and—when the grammar supports it—where its body and enclosing scope lie. A reference records a use site: a call, import, inheritance relation, read, write, documentation mention, or composition relationship. These records are deliberately less ambitious than a compiler’s semantic model. They preserve what the syntax can show before asking what a reference means.
Tree-sitter supplies the common parsing substrate. Each supported language contributes a grammar and a tags query. The query identifies definitions and references through a shared capture convention, while a small mapping translates capture names into the project’s internal roles. That lets one extraction engine cover the current set of 24 vendored grammars without writing an entirely separate AST walker for every language. The shared mechanism does not erase language differences; some grammars need filters or special handling where their syntax represents definitions as calls, places bodies beside rather than inside signatures, or cannot encode a runtime relationship in a static form.
The same approach extends beyond conventional programming languages. Configuration keys in JSON, TOML, and YAML can become section-like symbols, but they do not create call edges. Markdown headings become document sections whose spans run to the next heading of the same or higher level. Those choices let an agent retrieve the documentation and configuration that explain a code path without pretending that a table key or a heading is a function. The unit changes with the source, while the principle stays fixed: index a navigable fact at the granularity the syntax actually supports.
Ripwire’s internal durable unit is a symbol, not a file. A file is stored as an attribute of the symbol because a file-level graph would let large or widely referenced files dominate the importance calculation merely by being containers. Symbol nodes allow a later query to ask who calls one function, which definitions a reference may reach, or which body should be expanded. The file still matters for path filtering, imports, history, and context, but it does not replace the more specific unit that an agent needs to act on.
The project keeps the extraction result in a data model that separates facts from later interpretation. The ingest result holds the sorted file list, symbol table, references, per-file metadata, health information, and additional relation records. At this point, “the name run appeared at this location” and “this reference is inside parseRequest” may be known, while “this call reaches the run defined in module A” is not. That latter judgment belongs to graph construction. Keeping the boundary visible prevents the extractor from smuggling a guess into what looks like raw source evidence.
Reference roles also matter because they determine which later views may use the fact. A call can participate in the call graph and structural ranking. An import can describe a file dependency and help a resolver narrow candidates. A read or write can support a uses report or a non-local-state analysis. A documentation mention can connect an explanation to a code symbol without pretending that prose invokes a function. These are not interchangeable edges, and the ingest model preserves the distinction so a later verb does not quietly promote every relationship into control flow.
That separation is useful when the source is polyglot. A C++ header, a Python module, a Markdown explanation, and a YAML workflow may all be relevant to one agent task, but their names do not carry the same semantics. Configuration sections can be searchable without becoming executable nodes. A Markdown heading can delimit a useful body without becoming a caller. A language-specific capture filter can discard a syntactic “call” that is actually a declaration form. The common pipeline provides a shared shape for evidence while leaving the meaning of each role to the stage that can justify it.
Parallelism is allowed in the expensive part of this work, but it is constrained. Tree-sitter parsers are not thread-safe, so each worker uses its own parser. Files can be processed concurrently into per-thread result lists. The merge step then restores the canonical order before symbols and references are assigned. A parser worker may finish file B before file A; that timing must not decide which receives the lower identifier, which reference is emitted first, or which top-ranked cutoff is later selected.
This also explains why the model cannot treat final node numbers as permanent facts. Node identifiers belong to the current sorted file-and-symbol space, so the identity assignment is rebuilt deterministically for the current crawl. If the admitted files or extracted symbols change, the graph receives a new, explainable set of identities rather than inheriting positions from an earlier observation. Stable ordering is what makes that change legible instead of turning it into unexplained rank churn.
There are two important consequences for an agent. First, a missing graph relationship may originate before resolution: the file might be unsupported, skipped, unreadable, or syntactically degraded. Second, a present reference is not yet a proven edge. The extractor has made the evidence available to the graph stage, but it has not decided whether a same-named symbol in another file is the intended target. That restraint is what allows the later resolver to label ambiguity instead of burying it in a confident-looking map.
The front half of ripwire is therefore a chain of controlled reductions. It reduces a filesystem to an admitted corpus, source text to definitions and references, and parallel work to a stable ordered model. Each reduction saves context or computation, but none is allowed to become invisible. The next question is where the graph stage begins: how ripwire resolves a reference when syntax offers several possible definitions, and how those resolved or unresolved relationships acquire different kinds of rank.