How a workspace becomes searchable
For zvec-grep, indexed search quality is decided before a query is ever ranked. The important question is what the workspace scanner admits, what the extraction pipeline preserves, and how those pieces are fitted into model-sized fragments; understanding that path lets you tell a scope problem or representation limit from a genuinely poor match.
An index is therefore not a photocopy of a repository. It is a designed evidence surface. Some files are deliberately left out, some are represented as code entities or Markdown sections, and some fall back to ordinary text. The result can be more useful than a raw file dump because it preserves the structures that help a searcher recognize a relevant location. It can also be incomplete in specific, diagnosable ways.
The index begins with a scope decision
When zvec-grep builds or updates an index, it starts by scanning the workspace root. The scanner applies several kinds of rules before it creates a file record: hard exclusions, default ignored paths, repository ignore files, explicit include and exclude patterns, file-type filters, hidden-file policy, size limits, and a binary check. These rules are not cosmetic. They decide whether a file can contribute any indexed evidence at all.
Two paths are always treated specially: .git and the workspace’s own .zvec-grep state are not scanned as ordinary content. The scanner also ignores common dependency, build, cache, generated, localization, and log locations by default. Repository-specific ignore rules are applied as well. These defaults protect the index from being dominated by vendor copies, compiled output, generated maps, or temporary material that is rarely the best evidence for understanding a project.
The defaults are not absolute. Zvec-grep provides path and type controls so an operator can narrow a large workspace early or explicitly include material that matters. An index can be limited to source and documentation paths, restricted to a language, bounded by depth or file size, or configured with additional ignore rules. This is a useful distinction: an ignored directory is not necessarily unknowable. It is outside the default semantic scope unless the operator changes that scope.
The scanner also recognizes file kinds before extraction. The current implementation has structural paths for several common programming languages, Markdown section handling, plain text and structured data handling, and optional raster-image handling. A number of binary document, archive, compiled, media, and database formats are skipped before extraction. Images are different from ordinary text: they are excluded by default, must be selected explicitly, and require an embedding model that accepts image content. A multimodal model in the catalog does not make every image enter every index automatically.
Size limits add another boundary. Unless an explicit limit replaces them, the current defaults allow up to 1 MiB for code, 256 MiB for text, 16 MiB for structured data, and 10 MiB for images. The numbers matter less than the principle behind them: a large source file can be technically readable while still being unsafe or impractical to process as one indexing input. The scanner records oversized files as skipped rather than pretending that the index contains a partial semantic understanding of them.
Binary detection covers files that do not have a clearly blocked extension. The scanner samples the file and looks for null bytes or an excessive proportion of suspicious control characters. A file that fails this check is skipped before a text extractor tries to interpret it. Empty files are skipped too. These are different from indexing failures: a skipped file was not admitted to the current scan, while a failure means the system attempted a later preparation, embedding, or commit step and could not complete it.
That distinction changes how a missing result should be investigated. Imagine that a repository contains a generated JavaScript bundle, a source TypeScript module, and a PDF design note. An exact zg --rg search may still inspect the bundle or other workspace files according to its own scope, while the semantic index may omit the generated asset and skip the PDF. If an indexed query returns nothing about the design note, the first question is not whether the embedding model understood the note. The note may never have reached an extractor. If a source module was admitted but failed during embedding, index status and failure diagnostics answer a different question.
The same applies to hidden files, nested repositories, and symlinks. The scanner has explicit rules for when hidden material or a symbolic link may be followed, and it avoids treating an embedded repository as ordinary parent-workspace content unless the path is explicitly included. This protects the ownership of the workspace boundary. A semantic result should be interpretable as evidence from the scope that was selected, not as an accidental mixture of every reachable directory.
Structure changes what a fragment means
Once a file passes the scope gate, zvec-grep tries to preserve the kind of structure that will help a later query find and interpret it. This is the point at which “searchable” begins to mean more than “contains a matching string.” The extractor decides whether a fragment represents a code entity, a document section, or a bounded piece of plain text, and it keeps a source range so a result can lead back to the original file.
For code with an available grammar and language adapter, the code extractor walks a syntax tree rather than cutting the file at arbitrary line boundaries. It identifies entities such as functions, classes, interfaces, modules, aliases, and values according to the language-specific rules. Alongside the source text, it can preserve a symbol’s name and type, the breadcrumb of enclosing scopes, a signature, documentation, and relevant modifiers. A query about a class’s initialization path can therefore encounter the class and its scope as part of the indexed representation, not only as an accidental collection of nearby words.
Breadcrumbs solve a common ambiguity. A method named load can mean very different things depending on whether it belongs to an authentication service, a configuration object, or a document reader. The enclosing scope supplies context that the method name alone lacks. Signatures provide another anchor: two functions may share a name but differ in their parameters or role. Documentation and modifiers can make a public, asynchronous, or static entity more distinguishable from a private helper. These fields are not a separate knowledge graph, but they give lexical and vector retrieval more useful material to compare.
Source ranges are equally important for evidence. A search result needs to show not only that an entity was considered relevant, but where a person or agent can read it. The extractor retains line and offset ranges for entities and their windows. That lets the search layer return a focused passage while preserving a path back to the original source. The index can support discovery without becoming the permanent authority for the source text.
Large entities create a second problem. A class or function may be too long to embed as one model input, yet its header and overall shape may be exactly what a discovery query needs. Zvec-grep gives such an entity a major outline fragment and additional bounded windows. A structural outline can include a class’s members or the names of calls found inside a function. The subordinate windows remain grouped with the major entity, so a relevant window can resolve to a coherent public result instead of appearing as an unrelated top-level file fragment.
This design balances three competing needs. The outline gives the index a compact representation of the entity’s identity and shape. The windows retain body material that would disappear if only the outline were stored. The group preserves the relationship between them for result presentation. The compromise is useful for discovery, but it does not turn the extractor into full program analysis. A list of call names is not proof of a complete call graph, and a breadcrumb does not establish every dependency between modules.
Component files illustrate why source coordinates must be remapped carefully. For Vue and Svelte files, the extractor can locate JavaScript or TypeScript inside a script block, parse that block with the relevant language rules, and then map the resulting ranges back to the original component file. The indexed symbol can be structurally understood without losing the location a reader needs to inspect. If no usable script block or structural result is found, the file can fall back to plain text rather than silently disappearing.
Markdown receives a related but different treatment. The extractor scans headings, respects fenced code while looking for them, and builds sections with heading levels and breadcrumbs. A section about “Refresh behavior” is therefore distinguishable from another section with similar words elsewhere in a document. If a section is too long, the extractor creates an outline for the section and bounded windows for its body. A Markdown file without headings can still participate through plain-text chunks, but it no longer has the same section-level structure.
Plain-text fallback is a deliberate degradation path. The text extractor divides content into bounded line-oriented chunks, uses overlap when a boundary might otherwise separate related material, and tracks each chunk’s range. This is better than treating an oversized file as one impossible input, but it carries less meaning than a syntax-aware entity or heading section. A recognized language with no structural grammar may be searchable as text while lacking symbol metadata. That is the difference between being admitted to the index and being represented in the richest available form.
The representation then becomes embedding content. For code and Markdown, zvec-grep can prepend compact metadata such as the symbol type and name, scope, signature, modifiers, documentation, heading, and section scope before the content sent to the embedding model. Up to 25% of the configured chunk budget is reserved for this metadata, which means the source body receives less room when the metadata is substantial. This is a conscious trade-off: structural labels can improve discovery, but they consume part of the input budget.
The default chunk settings for code, Markdown, and ordinary text are 3,600 characters with 540 characters of overlap before metadata adjustment. Character counts are not token counts, so a fragment that fits a character budget can still exceed a model’s input limit, especially across languages or with dense code. The selected embedding model enforces its own token limit for each fragment. If the provider truncates an input, zvec-grep records the affected fragment count in file status rather than treating the result as indistinguishable from a complete embedding.
That accounting is useful because truncation can be quiet from a user’s perspective. The index may build successfully and return relevant results, while the tail of a long fragment contributed less—or not at all—to its vector representation. A model with a longer input limit can reduce that risk, and narrowing the indexed content can reduce it too. Neither choice guarantees a better answer in every repository; they change the boundary of what the embedding step can see.
The embedding model also filters what can be committed. A text-only model cannot embed an image fragment, while a model that accepts images may still be constrained by image size and explicit discovery policy. The model returns a vector for each accepted fragment, and the index stores that vector alongside the searchable text, metadata, and source range. If preparation, embedding, or storage fails, the file is marked accordingly so the operator can distinguish a failed path from an intentionally skipped one.
This full path explains why a semantic miss is not a single kind of failure. The file may have been excluded by scope, classified as unsupported or binary, flattened into a less informative fallback, split across a boundary that weakened its context, truncated by the model, or indexed correctly but ranked below other candidates. Zvec-grep’s evidence-routing model is only as strong as this upstream representation. Before changing a query, it can be more productive to ask whether the relevant material exists in the semantic index in the form the query assumes.
The practical lesson is modest but powerful: configure scope for the question, preserve structure where the project can support it, and treat fallback and truncation as visible limits. A source-linked result is valuable because it brings a reader back to the workspace. It is trustworthy only when the reader also understands which part of the workspace the index represents and how that representation was produced.