Where zvec-grep earns its keep—and where it does not

The public evidence for zvec-grep is narrower than the project’s product ambition. The useful question is not whether one benchmark makes the system “better,” but what a controlled study says about ranked workspace discovery, what it leaves out, and when exact search or current coverage still matters; that distinction supports a defensible adoption decision.

The project’s central claim is about the economics of finding evidence. Semantic discovery can surface a relevant entity before a person or agent knows its vocabulary, and compact source-linked results can reduce broad scans and repeated context loading. Those mechanisms should produce measurable gains on questions with paraphrased clues and large search spaces. They should produce smaller gains when a known identifier and exact matching already lead directly to the answer.

The retained evaluation is useful because it measures that claim with a paired design. It is also limited because it covers one fixed corpus, one agent and model configuration, one embedding setup, and a report produced by the project itself. Reading both sides of that boundary is more informative than repeating a headline percentage.

What the paired study measured

The BrowseComp-Plus study used 100 cases from a fixed corpus of roughly 100,000 human-verified documents and ran three independent trials per case for each profile, producing 300 paired trials. The Baseline and zvec-grep profiles held the case prompts, corpus, agent, model, reasoning settings, environment, and limits constant. The treatment profile added a prepared zvec-grep index, its MCP search access, and standard usage guidance. Index preparation was measured separately rather than charged to the agent-execution time.

That design addresses the most important comparison problem: a retrieval tool should not be credited for a stronger model, a different prompt, or a friendlier task set. The benchmark documentation explicitly warns that such changes would confound the result. It also requires a smoke test because a tool that is installed but unavailable can change an entire agent trajectory. In the reported study, treatment used zvec-grep in all 300 trials, with 988 successful calls and one failed call out of 989.

The primary results were:

The quality numbers are close. Baseline answered 98.67% of cases accurately, while zvec-grep answered 99.00%. Across the paired outcomes, both profiles were correct in 294 trials, Baseline alone was correct in two, treatment alone in three, and neither in one. That pattern does not support a claim that the retrieval layer universally transforms answer accuracy. It supports a more modest conclusion: in this study, adding zvec-grep maintained accuracy while reducing measured agent-search resources.

The efficiency changes are large enough to be operationally meaningful. Average input tokens fell by 37.56%, average tool calls by 43.52%, and measured agent time by 38.58%. At the case level, 67 of the 100 cases used fewer input tokens with zvec-grep, and the median case-level token change was a 25.76% reduction. The average reduction being larger than the median matters: a few expensive Baseline search tails were shortened substantially, while the median and the case win count show that the result was not produced only by one outlier.

The retrieval diagnostics explain the mechanism more directly than the headline totals.