From Raw Files to Progressive Context
OpenViking’s ingestion and loading pipeline is the focus here: how an imported document, repository, or media file becomes context that an agent can inspect at different depths. Following that path makes three things clear—the purpose of L0, L1, and L2; the difference between a directory’s semantic sidecars and its source files; and the boundary between deterministic parsing, asynchronous model work, and searchable readiness.
The three layers are decisions, not copies
When an agent receives a search result, the most useful next step is not always to open the full file. A directory may contain dozens of documents, only one of which concerns the current question. Loading all of them wastes input budget and forces the model to sort through detail before it knows whether the directory is relevant. OpenViking addresses this with three information layers.
L0, the abstract, is the shortest representation. It gives a quick account of what a directory is about and supports vector retrieval and early relevance decisions. A good L0 might say that a directory contains an API authentication guide covering OAuth, JSON Web Tokens, and API keys. It is a signal, not a replacement for the guide.
L1, the overview, is a broader map. It describes the directory’s scope, key ideas, and internal navigation, often pointing toward the files or subdirectories that matter. It supports reranking and helps an agent decide which parts of the directory deserve a deeper read. An overview can tell the agent that one file explains the OAuth flow, another covers token validation, and a third discusses API keys.
L2, the detail, is the original or fully parsed content. It is the level an agent reads when it needs exact wording, implementation detail, or evidence for an answer. L2 does not have one universal body limit because its purpose is fidelity to the source, although the caller and context assembler may impose practical budgets.
The ladder therefore corresponds to three decisions:
- Is this area relevant? Start with L0.
- What does this area contain, and where should I look? Use L1.
- What exactly does the source say? Read L2.
The default sidecar body limits make the difference concrete: L0 is limited to 256 characters and L1 to 4000 characters, with the limits configurable. These are limits on the Markdown bodies, not on the sidecar metadata. The values are not a promise that every response will consume exactly that amount; they are design constraints for compact derived views.
Why the summaries belong to directories
A tempting mental model is to imagine three summaries attached to every file. That is not how OpenViking describes the system. L0 and L1 are directory-level semantic sidecars, normally stored as .abstract.md and .overview.md. Ordinary files do not receive a matching pair of sidecars. Summaries of files are inputs to the overview of the directory that contains them.
This choice preserves hierarchy. Consider a repository with an authentication directory containing separate files for OAuth, JSON Web Tokens, and API keys. The directory overview can present those files as related parts of one subject. A search can first match the directory because its combined meaning is relevant, then descend to the file whose detail answers the question. If each file were summarized in isolation and returned without its parent relationship, the agent would have to reconstruct the subject’s structure from a loose list of hits.
The directory is also the natural unit for navigation. An L0 tells the system whether to consider a directory. An L1 tells the agent how to explore it. The files remain the evidence-bearing detail. This division avoids asking one representation to perform incompatible jobs: a two-sentence abstract should not be forced to serve as a navigation map, and a full source file should not be loaded merely to decide whether an entire area is relevant.
There are useful exceptions to keep in view. A directory can have an abstract without an overview—for example, a newly created directory may initially have only L0. A sidecar can also contain frontmatter describing its directory, import source, generator, and freshness, but normal abstract and overview accessors expose the visible body rather than that metadata. An agent that needs raw sidecar provenance must request the sidecar directly. These distinctions matter because “what the service stores” and “what it presents for semantic context” are not always the same view.
Parsing creates structure before meaning
The journey begins with a parser, not with a language model. OpenViking’s documented flow is:
The parser handles format conversion and structural organization. The documented routes cover Markdown, plain text, PDF, HTML, code, images, video, and audio. For a long document, smart splitting can keep a small document as one file, split a larger document by headers, merge short sections, and create subdirectories for very large sections. The purpose is to create a navigable tree before semantic enrichment begins.
That separation is important for both reliability and explanation. Parsing does not call an LLM. It can place the source material into a temporary structured tree even when a semantic provider is unavailable. TreeBuilder then identifies the document root, maps it to the target resource or user scope, moves the directory into content storage, removes temporary files, and submits semantic work. At this point the source can exist as stored content even though its derived summaries and embeddings are still pending.
This is the first boundary between “stored” and “searchable.” The source file is the durable object an agent must ultimately trust. L0, L1, and vector records are derived views that make that object easier to discover and load. An import task returning an accepted or completed state must therefore be interpreted in light of which downstream semantic work has finished, not only whether bytes have been placed somewhere.
Semantic processing works upward from the leaves
The semantic queue processes a directory tree bottom-up. It first generates or collects summaries for files in a leaf directory, then uses child-directory abstracts to build the parent overview. Within a directory, file summary calls can run concurrently within configured limits. The system writes the L1 overview, extracts the L0 abstract from it, stores the sidecars, and queues the relevant records for embedding.
The bottom-up order gives a parent a meaningful basis for summarization. A project directory can describe its documentation and source subdirectories because those children already have abstracts. A namespace root can be refreshed after lower levels change. The result is not a single huge model prompt containing every descendant file; it is a chain of compact derived descriptions that lets meaning travel upward without discarding the source tree.
The relationship between L1 and L0 is also deliberate. OpenViking documents L0 as being extracted from the L1 body: specifically, the brief-description paragraph after an H1 title and before the first second-level heading. This makes L0 a concise semantic view of the broader overview rather than an unrelated second summary. If a caller writes a body update to an existing sidecar, the stored metadata is preserved and the body is not immediately overwritten by regeneration; user-written sidecar bodies and model-generated semantic refreshes are different operations.
Vectorization follows semantic generation. The vector index stores references and metadata used for retrieval, while content remains in the filesystem layer. The embedding input for L0/L1 includes the visible body and an explicitly allowed directory field; generator and freshness metadata are not silently turned into semantic meaning. This keeps retrieval tied to what an agent can actually read as context, rather than to internal bookkeeping.
Code and media broaden the tree, but not the guarantees
OpenViking does not treat every source format as ordinary prose. For code, the documented summary pipeline first tries a maintained Tree-sitter query for the language. If one is unavailable, it uses the language-pack processing route. An LLM code summary is a fallback only when the extraction route produces no useful skeleton. The skeleton can expose imports, classes, methods, functions, and other language-level symbols, giving a directory overview a useful structural account without asking a model to summarize every line.
This matters for an agent investigating a repository. A question about where authentication is implemented may be better served by a class or function outline than by a vague paragraph about the file. The outline helps retrieval and navigation; the original code remains L2 and must be read when exact behavior matters. Structural extraction improves the route to evidence, but it does not prove that the implementation is correct or that a generated description captures every edge case.
Images, audio, and video follow a similar principle. A multimodal parser or configured vision-language model can first produce a text description, and that description contributes to the containing directory’s semantic views. OpenViking does not create a separate L0/L1 sidecar for each image, audio file, or video. A media leaf can still be stored and indexed, but the quality of its semantic representation depends on the configured model and on whether that provider understands the particular format.
The distinction between “ingestible” and “understandable” is especially important for media. A file may be accepted by the existing parser while the selected external provider cannot analyze it. In that case, the system may retain the file but have only limited or empty semantic description. Media processing can also send content to an external provider, whose retention, cleanup, and billing policies remain relevant. Multimodal support expands the kinds of context an agent can use; it does not remove provider dependency.
Freshness is part of the loading decision
A generated overview is useful only if the reader understands what it covers. OpenViking records freshness information about a directory’s direct children: how many entries contributed, how many were sampled, how many were not, and whether known child changes are still pending. When a directory has more direct entries than the configured sample limit, the system uses deterministic, order-preserving sampling so repeated refreshes of an unchanged tree do not produce needless changes.
That metadata does not make a stale overview unusable. It makes the staleness legible. An overview with pending child changes can still guide an agent, but a task that requires the latest state may need to read the source or wait for semantic refresh. This is a better contract than presenting every summary as current without qualification.
The maintenance cost is equally real. The current implementation attempts to bubble a successful resource or skill semantic task upward toward the namespace boundary, even when the newly generated child summary may not have changed. The project marks control of this bubbling frequency as future work because hot, deeply nested directories can suffer repeated refreshes and write amplification. Progressive loading reduces read cost, but it creates a derived-view maintenance problem on the write side.
The same pattern appears in queues and model calls. A VLM or embedding service can time out, rate-limit a request, or be unavailable. Retries and concurrency limits can help, but they do not turn external inference into a local deterministic operation. An agent may therefore encounter a source that is stored but lacks a fresh abstract, overview, or vector record. The correct response is not to pretend the tiers are complete; it is to treat semantic readiness as a state that can lag content storage and recover through later processing.
What the pipeline lets an agent decide
By the end of ingestion, OpenViking has not merely converted a file into a vector. It has attempted to create a layered route through a context tree. L0 offers a cheap relevance signal. L1 gives a directory-level map. L2 preserves the detail that can support an exact answer. Parsing establishes the structure; semantic processing adds model-assisted meaning; vectorization makes that meaning discoverable.
That route also explains the limits of the design. The summaries are derived, directory-oriented, and potentially stale. Code skeletons improve structural discovery but do not replace code reading. Media can be stored without being deeply understood. External model quality, queue health, and refresh policy affect what the agent sees. The payoff is not perfect compression; it is a disciplined choice about how much context to expose before deeper reading is justified.
The next problem follows naturally. Once an agent has a layered tree, it still needs a method for finding the right branch and deciding how much of it fits in the current request. That is where OpenViking’s retrieval system turns progressive context into guided exploration.