From Thumbs-Down to Root Cause
In NVInfo AI, a negative reaction is the beginning of an investigation, not the end of one. This chapter focuses on how the Adaptive Data Flywheel moves from a user's signal to a provisional explanation of failure, and why that movement requires more than a score. The practical understanding is a distinction among detection, attribution, labeling, and diagnosis: each step narrows the question, but none should be allowed to claim more certainty than its evidence supports.
That discipline matters because NVInfo AI is a chain of dependent decisions. A response can be unsatisfactory because the system chose the wrong expert, changed the meaning of the question during rephrasing, failed to retrieve an available document, ranked evidence poorly, generated a misleading answer, or failed to cite its support. A thumbs-down does not identify which of those happened. It tells the flywheel where to look.
A negative signal opens several hypotheses
The monitoring design combines direct feedback—thumbs-up and thumbs-down—with indirect behavior such as re-queries and session abandonment. It also preserves response and pipeline telemetry: the original query, the response, the knowledge category, the selected expert, time taken, the rephrased query, intermediate retrieval results, prompts, and guardrail metrics. These fields do not all answer the same question, and that is their value.
A thumbs-down is evidence about a user's experience. A repeated query is evidence that the interaction did not end the user's search, though it does not prove that the answer was wrong. A long response time is evidence about latency, not correctness. A missing citation may explain a loss of trust, even when the answer is accurate. The selected expert and rephrased query provide clues about earlier interpretations. Treating these signals as interchangeable would collapse several distinct hypotheses into one vague category called “bad answer.”
The first stage, detection, is therefore deliberately modest. It identifies an interaction or a pattern worth examining. Detection asks, “Where should attention be spent?” It does not yet ask, “What is the correct repair?” That separation is especially important when feedback is sparse or uneven. The paper reports 495 negative samples from thousands of users and notes that participation was limited. Negative outcomes may be reported more often than positive ones, so the observed set can reveal important patterns without being a representative sample of every query.
Analysis adds context to the signal. Suppose a user asks a policy question and receives an inability-to-answer message. Looking only at the final text suggests a generation problem. Looking at the trace may show that the request was routed to a holiday specialist, so the relevant policy material was never placed on the path. The trace does not prove that routing was the only cause, but it changes the set of plausible explanations.
The system's privacy boundary makes this investigation harder. The case describes requirements to remove personally identifiable information and restrictions on storing complete query-response pairs under enterprise policy, GDPR, and CCPA. A minimized record can preserve the selected expert, timing, rephrasing, and retrieval structure while omitting content that should not be retained. That is a necessary protection, but it means that some contextual evidence may be unavailable later. Absence from the retained trace is not evidence that an event never occurred; it may be the consequence of what the system was allowed to keep.
The route can be the cause of the answer
The vacation-policy example shows why attribution must follow the pipeline's direction. The user asked, “How many vacation days does NVIDIA Canada have?” NVInfo AI sent the query to a Holidays Expert instead of a Policies Expert, and the resulting response said there was not enough information to answer.
There are at least two different interpretations of that outcome. Under the first, the knowledge base lacks the required Canadian policy. Under the second, the policy exists but the system entered the wrong domain path. The response alone cannot distinguish them. The selected expert is the discriminating trace: it reveals an upstream decision that could prevent the relevant material from being retrieved at all.
This is attribution. It asks, “Which stage most plausibly produced the failure?” The answer should be more specific than “the AI was wrong,” but it may remain provisional. A wrong route can coexist with a retrieval problem. The correct expert might also fail to locate a document. The response generator could still mishandle correct context. A good diagnosis does not erase competing causes simply because one attractive explanation has appeared.
The direction of the pipeline creates a useful test. If the route changed, would the intended knowledge source become available? If yes, routing is a credible contributor. If the relevant document was available to the selected expert and still absent from retrieval, the analysis should move downstream. If the document was retrieved and the answer misrepresented it, generation or citation becomes more plausible. The test does not require assuming that one layer is always responsible; it asks what evidence would distinguish the alternatives.
This is why an answer score by itself is inadequate. A score can tell a team that a result was poor. It cannot show whether the error happened before search, during search, or after the needed evidence had already been found. The trace turns a symptom into a sequence.
Rephrasing can change what the system is able to find
The second NVInfo example involves the query “RESS planning team.” The system incorrectly expanded “RESS” as “NVIDIA Resource Planning team” instead of “Real Estate & Site Services.” The failure then appeared as an inability to retrieve the intended department information.
Here the visible problem is not simply that retrieval missed a document. Search received a distorted representation of the user's intent. A retriever can perform faithfully against that representation and still return the wrong material or nothing useful. The rephrased query is therefore a critical diagnostic artifact. It lets an analyst compare what the employee meant with what the search stage was actually asked to find.
The two examples reveal different causal shapes. The vacation question retained its words but crossed the wrong domain boundary. The RESS question encountered a semantic transformation that changed the meaning of an acronym. In both cases, the final answer is downstream of an earlier interpretation. That is the practical reason to examine intermediate states: they expose decisions that a final response cannot explain.
The distinction also protects against a tempting but harmful repair. If the team sees an unanswered response and adds more answer-generation instructions, it may make the generator more fluent without giving it the missing policy or the correct department context. If it sees an acronym problem and changes retrieval ranking, it may optimize the wrong search input. Diagnosis should precede remediation because a change in the wrong layer can add complexity without restoring the missing evidence.
What the percentages do—and do not—say
The case reports 26 routing errors among 495 negative samples, or 5.25 percent of that sample. It also reports query-rephrasal errors at approximately 3.2 percent, based on analysis of 250 of the 495 samples and about 10 incorrect rephrasals. Elsewhere, the paper represents the rephrasal estimate as approximately 16 out of 495. The difference is not a reason to discard the finding, but it is a reason to retain its qualifiers.
The routing count is an explicit count within the reviewed negative-feedback set. The rephrasal figure is an extrapolation from a manually examined subset, and the paper presents it approximately. Neither number is a measured error rate for all NVInfo AI queries, all employees, or all enterprise assistants. Nor does the remaining “other errors” category mean that every case in it was correctly attributed to a different mechanism. It includes what the selected analysis did not classify as the two highlighted targets.
Denominators are part of meaning. “5.25 percent” without its denominator sounds like a population prevalence. “26 of 495 negative samples” tells the reader what was actually observed. The difference matters for planning: a small category can still justify a targeted fix if its impact is high, while a large unclassified category may signal that the monitoring or attribution process is incomplete. A rate can prioritize an experiment; it cannot alone establish that the experiment will improve the whole system.
The same caution applies to the reported rephrasal examples. Finding ten incorrect rephrasals in the reviewed subset supports the existence of a recurring pattern. It does not show how many unobserved acronyms, locations, or domain-specific terms fail in production. A flywheel that learns only from the cases users choose to report may improve the visible edge of the problem while leaving silent failures untouched.
From automated triage to a defensible diagnosis
The case combines manual analysis with automated classification. An automated classifier can make detection and candidate triage faster: it can scan traces, group similar failures, and flag interactions whose route or rephrased query looks inconsistent with the domain. That is a useful division of labor. Machines process more candidates; people or source-backed checks decide whether the candidate's interpretation is correct.
The boundary is not that automation must be absent. It is that an automated label is still an interpretation of the evidence. If a model classifies another model's route as wrong, the classification may be accurate, but the model being judged cannot be the only authority that proves it. A domain expert can compare the route with the actual subject of the question. An enterprise document or policy can establish which answer and citation should have been available. Without that external check, the flywheel risks turning a plausible classifier output into a training label simply because it is convenient.
Manual review is costly, and it can itself be inconsistent. That does not make it optional whenever a label changes the model. It means the workflow should reserve human or source-grounded effort for the transitions where a candidate becomes a correction, and should make uncertainty visible for cases that do not survive review. A binary “routing error” label may be enough to find a cluster; it may not be enough to understand whether two similar-looking queries require the same repair.
The most useful output of diagnosis is therefore not a confident story but a bounded statement: this interaction was negatively rated; the trace shows a particular route or rephrasing; the domain evidence supports a different interpretation; and a targeted remediation is justified for that pattern. Each clause has a different evidentiary basis. Keeping them separate makes later evaluation possible.
The sequence makes ownership visible. The employee supplies an interaction and a reaction. NVInfo AI produces the response and trace. The classifier proposes a candidate, but the domain reviewer checks whether the proposed explanation fits the underlying evidence. The improvement plan receives a bounded diagnosis rather than an unexplained score. The diagram is not a claim that every case follows this exact synchronous order; it represents the logical handoffs described by the monitoring and analysis design.
The diagnosis remains provisional until a reference can disagree
The two NVInfo failures demonstrate why the flywheel needs an external reference even before training begins. For a policy question, the reference may be the relevant policy content and its expected citation. For an acronym, it may be domain knowledge that identifies “Real Estate & Site Services.” The user's negative rating helps select the case. The trace helps locate the suspected stage. The reference determines whether the interpretation is actually wrong.
That reference does not have to answer every possible question automatically. It has to be capable of rejecting a convenient explanation. If a route is changed because an automated classifier prefers another expert, a representative set of domain questions should test whether the new route preserves the intended boundaries. If a rephraser is changed because one expansion was wrong, examples with locations, acronyms, and conversational context should test whether meaning survives the transformation.
This is the point at which diagnosis becomes a candidate for remediation, not yet proof of improvement. The flywheel should remember what was observed, how it was attributed, which external evidence supported the correction, and what remained unknown. A later update can then be evaluated against the same kind of reference rather than against the model's own revised output.
A thumbs-down is valuable precisely because it is incomplete. It interrupts the system's assumption that the response was good enough. But the interruption must lead to a chain of increasingly grounded questions, not to an automatic rewrite. The next challenge is to decide which artifacts in that chain count as evidence, which merely suggest a direction, and where an independent reference must be able to say no.