From Files to a Graph: The Two-Speed Extraction Pipeline
Graphify's central engineering problem is how to turn unlike source material into one navigable structure without pretending that every input can be processed in the same way. This chapter follows Graphify from discovery through extraction, graph construction, analysis, and reporting, with particular attention to the boundary between deterministic local structure and model-backed semantic enrichment. The practical result is a way to predict where a graph result comes from, where cost or uncertainty enters, and what kind of refresh a changed source will require.
The pipeline is easier to understand as a set of transformations than as a command that “indexes a folder.” A corpus first becomes a classified set of inputs. Those inputs become fragments of nodes and relationships. The fragments are merged, resolved, clustered, and analyzed into artifacts that a person or assistant can query. At each boundary, the system changes not only the representation of the data but also the questions the reader can responsibly ask.
The pipeline starts by deciding what counts
Before Graphify can discover relationships, it must decide which files belong to the corpus and how they should be treated. Its discovery stage classifies inputs into broad families such as code, documents, papers, images, office files, and video or audio. That classification is not cosmetic. It selects the extraction path, determines whether an optional dependency is needed, and predicts whether the result will contain structural facts, semantic descriptions, or a locally generated transcript.
Discovery also applies the practical boundaries that make a repository scan usable. Ignore rules prevent generated output and intentionally excluded paths from becoming part of the map. Noise directories and sensitive-looking files are screened so a project scan does not casually turn credential stores or private key material into graph content. The discovery layer performs corpus-health checks as well. A very small corpus may not need a graph for token reduction, while a very large corpus may need warnings about resource use. Those warnings are not a verdict that Graphify is appropriate or inappropriate; they tell the operator that the economics and expected value have changed.
This first stage therefore owns the admission boundary. A file that is ignored is not a missing relationship. A file whose format requires an optional extra is not evidence that the underlying corpus has no content. A parser that returns no nodes is surfaced as a problem rather than silently treated as a successful empty result. These distinctions matter because the graph is a derived view: absence from the graph can mean “not in scope,” “not supported,” “failed to parse,” or “not present in the source,” and those causes should not be collapsed.
The boundary also determines who owns the next state transition. Discovery owns the list of candidates and the explanation for excluding them; the extractor owns the transformation of an admitted file into nodes and edges; the builder owns the identity of the combined graph. If a document is newly added, its content must enter through the semantic path. If a source file changes, the structural result for that file and the cross-file relationships affected by it may need to be rebuilt. If a dependency is missing, the right state is “not extracted yet,” not “extracted and empty.” This sounds like bookkeeping, but it is what prevents downstream analysis from turning an operational omission into an architectural conclusion.
The same logic explains why discovery includes health signals instead of simply walking every readable path. A repository can contain generated artifacts that would swamp the meaningful graph, copied dependencies that duplicate entities, or files whose names suggest secrets rather than source. A scan that admits all of them may be technically complete while becoming less useful and less safe. Conversely, an aggressive exclusion rule can remove real project material. Graphify's discovery stage is therefore a policy boundary with consequences for recall, privacy, and later interpretation; it is not just a prelude to the “real” extraction work.
Once the input set is defined, the pipeline separates the work into structural extraction and semantic enrichment. The structural path is designed for source code. Graphify uses tree-sitter-based parsing to identify entities such as files, classes, functions, imports, calls, and types, along with source locations. The code is read locally; a code-only corpus can be processed without sending the code to a language model. That local boundary is one of the tool's clearest design choices: the first description of a codebase is generated from its syntax and relationships rather than from a remote interpretation of its entire text.
The code path itself has two conceptual passes. First, each file is parsed into a local set of nodes and edges. Then cross-file resolution connects definitions and uses that cannot be understood from one file in isolation. A written import statement is an explicit structural fact. A link from a call site to the one plausible definition elsewhere may be a resolved inference. The second pass is what turns separate file fragments into a codebase-level map, but it is also where the graph must preserve uncertainty. More connectivity is useful only when the reader can tell whether the connection was directly written or reconstructed.
The repository supports a broad and changing set of code formats, but its own retained documents disagree about the exact language count: one description says 37 tree-sitter grammars, while another says 25 languages, and the current dispatch and dependency files enumerate still more categories. The defensible conclusion is that Graphify targets dozens of code formats, not that one exact count is stable across the project. The disagreement is itself a reminder that a pipeline description has a version and a source, just as a graph does.
The structural path can run in parallel for sufficiently large batches, and its results are cached by content so unchanged files need not be parsed again. Those are important operational details, but they serve a conceptual purpose: Graphify treats structural extraction as a repeatable transformation of source content. When the source is unchanged, reuse is possible. When it changes, the affected structural view can be rebuilt without paying the cost of reprocessing every file or involving a model merely to recover syntax.
There is a second reason to keep the local pass explicit: it provides the context used to resolve relationships across files. A changed caller may need to be matched against an unchanged callee, and an incremental run must be able to consult the retained structural context without claiming that the unchanged file was freshly parsed. This is a useful division of responsibility. The current run owns the new extraction; the prior graph supplies read-only context for resolution. When that context is missing or an extractor fails, a smaller graph is not automatically a valid replacement for the previous one.
The local path therefore has a failure mode that is different from a model hallucination. A grammar can fail to load, a source file can contain syntax the extractor cannot recover, or an accepted file can yield too few nodes. The implementation surfaces these conditions and avoids caching some anomalous empty results so a later run can retry. The operational lesson is simple: successful process completion is not the same as complete evidence. A graph build should be judged by what was admitted, extracted, and retained, not only by whether a command returned a report.
Two speeds, one graph
Documents and media need a different treatment because their useful relationships are often semantic rather than syntactic. Graphify's documented flow transcribes video and audio locally, while markdown, PDFs, images, and transcripts can pass through parallel assistant or model subagents. The exact backend may vary, but the boundary is stable: a semantic pass is interpreting content that does not expose a complete call graph through a programming-language grammar.
That difference creates both capability and cost. A code parser can see that one function calls another because the relation is represented in syntax. A semantic extractor can connect an implementation concept to an explanation in a document, a paper's idea to a code component, or two differently worded concepts that belong in the same discussion. Such links can make a mixed corpus much more useful than a code-only map. They can also involve model calls, token cost, and a data path outside the local machine. The common graph makes the outputs interoperable, but it does not make their origins interchangeable.
The pipeline's next job is to merge these outputs into a coherent graph. Each extractor produces a structured fragment rather than a finished report. The build stage combines node and edge records, normalizes identifiers and source paths, reconciles duplicates, and preserves the relationships that point into the combined corpus. Deduplication is not merely tidying. If a code parser and a semantic pass describe the same file or entity with slightly different identifiers, leaving both records in place can split the evidence into ghost nodes. If unrelated same-named entities are merged too aggressively, the graph invents a relationship by erasing a boundary. A useful build therefore has to be conservative about identity while still joining equivalent records.
After construction, Graphify can analyze the graph as a structure rather than only as a collection of extracted facts. Community detection groups nodes by connectivity, producing a view of subsystems and boundaries. Other analyses identify highly connected real entities, cross-file or cross-community connections, and questions that deserve investigation. A report then combines corpus checks, graph statistics, confidence proportions, community navigation, and these analytical highlights. Exports make the same derived structure available in different forms, including a browser-oriented visualization or machine-readable graph data.
These downstream stages do not create new source truth merely by rearranging the graph. Clustering can reveal that two sets of nodes are densely connected, but it cannot prove that the project maintainers intended them as one subsystem. A high-degree node may be central to the domain, or it may be a mechanically generated file hub. A surprising connection may be a valuable cross-boundary discovery, or an inferred name match that deserves suppression. The analysis layer is most useful when it changes the order of investigation while leaving the evidence label attached.
The report is consequently an audit trail as much as a presentation. It can show how many relationships were extracted or inferred, which communities were found, what the corpus check reported, and which connections or questions deserve attention. That makes it possible to compare the map with the reader's experience of the repository. If a key subsystem is absent, the reader has a place to ask whether discovery excluded it, extraction failed, or a relationship was not represented. If the report looks unexpectedly dense, the reader can ask whether a utility hub or duplicate identity is inflating the topology.
The important causal sequence is therefore:
- Discovery defines scope. It decides what is in the corpus, what category each input belongs to, and which failures or exclusions must remain visible.
- Extraction creates evidence fragments. Local parsing records structural facts; local transcription or model-backed semantic work supplies other kinds of content.
- Building creates identity and connectivity. Fragments are merged, duplicates are reconciled, and cross-file relationships become navigable.
- Analysis supplies orientation. Communities, central nodes, surprising links, and open questions turn topology into a human-readable view.
- Reporting and querying expose the result. The graph becomes something a reader can inspect, traverse, and compare with the source.
The diagram shows a convergence, not an erasure of differences. The three source paths meet at graph construction, but the provenance of their results remains important. A transcript-derived concept has a different chain of evidence from a tree-sitter-extracted function. A model-generated relationship has a different failure surface from a written import. Graphify's later confidence labels and audits exist because the graph needs to carry those distinctions forward after the paths converge.
This is also why the pipeline cannot be judged only by its successful final report. A missing optional grammar, a malformed or unsupported file, a zero-node parse, an incomplete semantic response, or a stale cached result can all produce a graph that looks coherent while omitting something important. The implementation includes warnings and retry or invalidation paths for several of these cases, but the reader still needs to interpret an apparently absent entity in context. “Not shown” is not automatically “not present.”
The two-speed design is ultimately a way to spend different kinds of effort where they have the most leverage. Deterministic structure gives the graph a stable local skeleton at low privacy and token cost. Semantic enrichment adds connections that syntax cannot supply, at a higher cost and with a more conditional trust model. Building, clustering, and reporting make the combination useful, but they do not flatten the distinction between observation and interpretation.
That distinction is the next necessary step. Once the pipeline has produced a graph, the key question is no longer how the files entered it. It is what each edge is claiming, what evidence supports that claim, and how a reader should act when the graph says “inferred” or “ambiguous” rather than “explicit.”