From Source to Structure

Understand Anything's first job is not to explain a repository; it is to establish what is there. This chapter follows the deterministic path from project inventory to structural graph, so you can tell which relationships come directly from files and which are added later as interpretation. That distinction gives the resulting map a stable surface: before asking what a component means, the system must know which files exist, which language rules apply, and where the observable relationships are.

Why the inventory comes first

A codebase is not a bag of independent files. An import only becomes meaningful when the target is resolved against the same project inventory as the source. A function only becomes useful as a graph node when its file and line range are known. A configuration file or deployment manifest may not import application code in the usual programming-language sense, but it can still shape what the application does. The scanner therefore establishes a shared frame before the more detailed analysis begins.

The current scanner prefers the repository's tracked and untracked working files as reported by Git, then has a recursive-walk fallback for directories where Git cannot provide the inventory. It produces stable, project-relative paths rather than embedding a developer's machine layout into the result. It also applies a layered ignore policy: built-in exclusions cover common dependency directories, build outputs, binary assets, lock files, editor data, generated artifacts, and other files that would add noise; project-level rules and command-line exclusions can narrow the scope further. A negation can bring a deliberately included file back into consideration.

This is more than housekeeping. An incomplete inventory changes the meaning of the graph. If a dependency directory is scanned as if it were project code, the graph is flooded with implementation details the team does not own. If a relevant configuration or documentation file is omitted, the graph loses context that could explain an otherwise mysterious behavior. If paths are handled inconsistently across operating systems, later stages may create edges to files that appear not to exist. Deterministic enumeration and normalization reduce those sources of ambiguity before a model is asked to reason about purpose.

Each retained file receives a language and a category. The language is inferred from extensions and recognized filenames: a TypeScript file, a Python module, a Dockerfile, a Markdown document, and a workflow file enter the pipeline with different expectations. The category is deliberately broader than “code.” Files can be treated as code, configuration, documentation, infrastructure, data, scripts, or markup. That distinction lets a YAML workflow be understood as a pipeline, a SQL file as a schema or table source, and a Markdown guide as a document instead of forcing every artifact into a source-code-shaped box.

The scanner also records line counts and a content digest. Line counts help later stages describe the size and complexity of a file and give the reader a rough sense of scale. The digest is a compact identity for the inventory's content, not a semantic judgment. If the same paths and bytes are encountered again, the scanner can recognize that the input has not changed. If the bytes change, the digest tells maintenance logic that further comparison is necessary; it does not by itself say whether the change affects graph structure.

The same scan prepares an internal import map for supported languages. Rather than making every file-analysis batch rediscover the whole repository and guess which relative path an import refers to, the project resolves confirmed internal targets once and passes the relevant slice to each batch. External packages are not treated as project files. This keeps the graph focused on relationships inside the analyzed boundary and gives later batching a complete topology to work with.

Consider a small path through the scanner. If an entry file imports an authentication module and a configuration file, the inventory first confirms that those two targets are real files in the project. The import map can then record those internal paths before any model sees the file. Later structural extraction may discover that the authentication module defines a login function and that another callable invokes it, but it does not need to guess whether an external package named auth belongs to the repository. The result is a cleaner separation between “this file points to that project file” and the later, more interpretive question of what the authentication path is for.

The inventory also gives failures a place to land. A permission error or a file that disappears during scanning can be reported as a skipped file rather than silently becoming an empty node. A file that is intentionally ignored is different from one that could not be read. Those distinctions affect both the completeness of the graph and the decisions made on a future run. A reliable map records not only what it found, but also the conditions under which something was not structurally examined.

flowchart LR A["Project files"] --> B["Inventory and ignore policy"] B --> C["Language and file category"] C --> D["Line counts and content digest"] B --> E["Resolved internal import map"] D --> F["Structural extraction"] E --> F F --> G["Canonical graph nodes and edges"] C --> H["Unsupported parser"] H --> I["Structural gap reported; semantic analysis may continue"]

The inventory is therefore the first contract between the repository and the graph. It says what is in scope, how each artifact should be treated, and which file-to-file relationships can be established before semantic analysis. It also creates the conditions for repeatability: later phases are not each working from a slightly different notion of the project.

A shared structural interface for many languages and artifacts

Once the inventory exists, Understand Anything selects an analyzer through a language registry. The registry maps language identifiers, file extensions, and recognized filenames to configuration. A language configuration can describe its display name, extensions, file patterns, notable concepts, and, where available, a Tree-sitter grammar. A framework registry performs a related job for frameworks detected in manifests. The important architectural choice is that these registries feed a common graph pipeline; adding language-specific knowledge does not require inventing a new graph format for every language.

This registry also makes file treatment more precise than extension matching alone. A filename such as a Dockerfile can select an infrastructure-oriented parser even though it has no conventional extension. A Markdown file can be assigned document semantics, while a YAML file under a workflow directory can be recognized as infrastructure rather than ordinary application configuration. The same project can therefore contain multiple languages and artifact kinds without forcing the reader to learn a different data model for each one. The registry chooses the extractor; the graph keeps the resulting identity and relationship vocabulary consistent.

For supported programming languages, the Tree-sitter plugin loads WebAssembly grammars and reuses parsers across files. Tree-sitter produces a concrete syntax tree: a structured representation of the source that makes declarations and relationships easier to locate than ad hoc text matching. Language-specific extractors then identify functions, classes, imports, exports, and call sites. The resulting records include names, parameters, owners where available, and line ranges. A call graph records that one callable invokes another when the extractor can observe that relationship; it does not claim to know every runtime dispatch that might occur.

The plugin is config-driven but not blindly generic. It loads only the grammars and extractors that are registered and available. A missing grammar or unavailable extractor is treated as a capability gap, not as evidence that the file has no structure. The normal analysis path can return an empty structural shape for an unsupported file so the broader pipeline remains resilient, while the strict path distinguishes unsupported analysis from a parser that failed after claiming to support the language. That distinction matters for incremental updates: uncertainty should lead to conservative reanalysis, not a false assertion that nothing changed.

Non-code parsers use the same structural interface with different outputs. The Markdown parser recognizes headings and local file or image references. YAML and JSON parsers expose top-level structure. A Dockerfile parser identifies build stages, ports, and instruction steps. SQL parsing can surface table, view, and index definitions with their fields. GraphQL parsing identifies types and query, mutation, or subscription endpoints. Other built-in parsers cover environment files, TOML, Protobuf, Terraform, Makefiles, and shell scripts. These are intentionally lightweight structural readings. They expose shapes that matter to the project graph without pretending to be complete interpreters for every DSL.

That common interface is what allows infrastructure and documentation to participate in the same cognitive model as application code. A deployment file can have a service node and a deployment relationship. A schema can have a definition node. A workflow can have pipeline steps. A Markdown file can document a code component. The graph does not need to understand every sentence or instruction at this stage; it needs enough faithful structure to anchor later explanations and show how different artifacts connect.

The interface also supports graceful degradation. If a repository contains a language with no registered grammar, the file is not necessarily discarded from the entire analysis. The semantic agent can still read it, produce a file-level explanation, and attach it to the graph, while the missing function/class/import structure remains a visible limitation. The distinction is important for a mixed-language project. A graph that contains a summarized file with no structural children is less detailed than one parsed by Tree-sitter, but it is more honest and often more useful than silently excluding the file.

How observations become graph relationships

The shared graph builder translates structural output into canonical nodes and edges. A file-level node records the file's project-relative path, summary metadata, tags, and complexity. Significant functions and classes become child nodes with their own names and source line ranges. A containment edge connects the file to each child. File imports become directed imports edges. A detected callable relationship becomes a directed calls edge between function nodes. The direction and edge type are part of the meaning; “A imports B” is not interchangeable with “B imports A,” and neither is the same as “A calls B.”

Canonical identity is what makes those relationships durable. The builder derives identifiers from a node's type, path, and name rather than from an arbitrary display label. A function called run in one file remains distinct from another function with the same name elsewhere. A source range provides an anchor for navigation and later comparison. When the dashboard selects a function or class, it can use that anchor to show the relevant code. When an incremental update replaces a file, maintenance logic can compare the old and new symbol identities rather than treating the entire graph as an undifferentiated blob.

The builder also guards against duplicate edges. An import relationship is emitted once for a source-target pair, and the graph records known weights for different relationships: containment is stronger than a broad dependency, while an import and a call express different kinds of confidence and scope. Weights are not probabilities that the code will behave a certain way. They are a normalized signal for graph consumers and visual layout, allowing more consequential or direct relationships to stand out.

The structural graph is consequently a map of observable organization. It can establish that a project contains a file, that the file defines a class, that one file imports another, and that a parsed function calls a known function. It can represent documentation, configuration, services, schemas, and other non-code structures when the corresponding parser finds them. It cannot, from this stage alone, establish why a team chose that architecture, whether a function's name accurately describes its business purpose, or whether a dynamic runtime path will occur in production.

This boundary is not a weakness to conceal. It is what lets the next stage add value without rewriting the facts. A parser is good at locating declarations and relationships that have a stable syntactic representation. It is not good at explaining the role of a module in ordinary language. Conversely, a fluent model can propose a useful explanation, but it should attach that explanation to a known file, symbol, or relationship rather than float free of the source. The structural layer supplies those attachment points.

When a parser cannot handle a file, the correct result is also specific. The system should not turn an unsupported language into a confidently empty architecture, and it should not infer that a deleted or unreadable symbol never existed. Current extraction outcomes distinguish a file that was skipped because no analyzer is registered from a file whose selected parser failed. Warnings and strict validation preserve that difference for later decisions. A file may receive semantic treatment, but readers should know that its structural neighborhood is incomplete.

The deterministic phase therefore ends with a deliberately modest achievement: a project-wide skeleton. It is stable enough to support identifiers, edges, layers, batching, navigation, and future comparisons. It is rich enough to include more than imports, but not so ambitious that it pretends syntax is intent. The next question is how Understand Anything fills that skeleton with explanations of purpose, architectural role, and learning order—while keeping those explanations visibly different from the source-backed structure they describe.

References