The First Checkpoint: What Counts as Evidence?

The Adaptive Data Flywheel around NVInfo AI has now given us two things that are easy to confuse: a path through which an enterprise agent produces and records an answer, and a method for turning a poor interaction into a likely diagnosis. This checkpoint connects those ideas into one practical question: what does each artifact prove, what does it only suggest, and where must an independent reference be able to reject the model's preferred explanation? The answer determines whether a faster local improvement is also a trustworthy one.

The key is not to divide evidence into “human” and “machine” and stop there. Different artifacts play different roles. A user reaction detects an issue. A trace helps attribute it. A domain expert or source-backed comparison can correct an interpretation. Synthetic data can expand a corrected pattern. A regression set can test a candidate change. Treating those roles as interchangeable is how a signal quietly becomes a label and a label quietly becomes the next definition of reality.

An evidence ladder, not a single score

At the bottom of the ladder is the production interaction. An employee asks a question, receives an answer, perhaps follows a citation, and gives a reaction or tries again. This is the closest evidence to actual use, which makes it valuable for discovering problems that a prelaunch test set did not anticipate. It is also uneven. The NVInfo case received negative feedback from only a portion of the people using the system, and users did not always explain the reason for dissatisfaction accurately. The interaction is evidence that a situation occurred; it is not automatically evidence of the correct answer.

The next rung is the trace. NVInfo records fields such as the selected expert, the rephrased query, retrieval results, timing, citations, and guardrail metrics alongside the response. A trace adds causal structure. It can show that a question about vacation days went to a holidays specialist, or that an acronym was expanded before search. It makes a failure more diagnosable than a final score alone. But a trace is still a record of system behavior, not an independent statement of what the system should have done.

Automated triage sits above the raw trace. A classifier can group interactions, flag suspicious routes, and make a large candidate set manageable. This is where model assistance is useful: it turns a high-volume search problem into a smaller review problem. Its output remains a hypothesis about the trace. If the classifier labels a route as wrong, that label can prioritize a human review; it does not prove which expert should have received the query.

Correction is the first rung at which the question of authority becomes explicit. A subject-matter expert can compare the query with the domain boundary. An enterprise policy or source document can establish the answer and the citation that should support it. In the routing example, the relevant distinction is not merely that the final response was unhelpful; it is that the vacation question belonged with policy information rather than the holidays path. In the rephrasing example, domain knowledge identifies “RESS” as “Real Estate & Site Services,” not “Resource Planning.” The correction is grounded in something outside the model's unverified output.

Only after a pattern has been corrected does synthetic expansion become well-founded. The NVInfo team used human-reviewed rephrasing examples and SharePoint content to generate approximately 5,000 examples.