Designing a Flywheel with an External Check
The evidence from NVInfo AI and the model-collapse experiments points toward a specific operating design. An Adaptive Data Flywheel should let models accelerate triage and data expansion, but it should assign authority for labels, coverage, evaluation, and release to references that the model being improved did not create on its own. This chapter applies that rule to the Monitor, Analyze, Plan, and Execute loop around NVInfo AI. The reader's practical task is to see where each kind of evidence belongs and where a candidate update must still be allowed to fail.
This is not a demand for a manual process around every query. The NVInfo case would not have been useful if human review had to write every possible rephrasing or inspect every interaction in full. It is a demand for role separation. Automation can make the path wider and faster; it should not make the path self-approving.
Give each artifact one job
The first design choice is to stop calling every record “training data.” A production interaction is valuable because it shows what employees actually asked and how the live agent behaved. A thumbs-down, a retry, or an abandoned session is useful for finding a candidate failure. A selected-expert field, a rephrased query, and retrieval trace help explain where the candidate may have gone wrong. These artifacts belong in monitoring and analysis because their first job is discovery.
A label becomes more consequential when it changes the model. In the NVInfo routing work, the team used production material to find candidate errors and combined original examples with corrections identified through an LLM-as-a-Judge process. The reported routing set contained 729 original positive examples and 32 corrections from 140 candidates, yielding 761 data points and 685 unique samples after deduplication, split 60/40 for training and testing. Those figures show how a small number of reviewed cases can make a focused intervention possible.
They do not turn the judge into an independent oracle. The case reports a 77 percent rate for detecting routing errors, but the supplied material does not describe a separately audited human gold set for all 140 candidates or provide full annotation agreement. The judge's output is therefore best treated as candidate selection or label assistance. An SME or a source-backed comparison must authorize the interpretation when it becomes the basis for a model update.
The difference is subtle but operationally decisive. The judge can ask, “Which of these traces look inconsistent?” A domain reference must be able to answer, “Which route actually fits the question?” The former scales attention. The latter establishes the target. If the same model is allowed to do both without a separate check, confidence can be mistaken for correctness.
The rephrasing work follows the same pattern. The team manually examined 250 SharePoint-related negative-feedback examples and identified ten incorrect rephrasals. It corrected examples such as a Netherlands payday query, then used a small set of corrected examples and SharePoint content to generate approximately 5,000 original/rephrased query pairs. Synthetic expansion increased coverage around a known problem. It did not make the generated pairs equivalent to 5,000 new employee interactions.
That distinction tells us how to use synthetic data responsibly. Generated examples can explore variations, expose a model to different wording, and reduce the burden of collecting sensitive real conversations. They should remain marked as generated, inherit the limitations of their seed corrections, and be tested against a source or human-grounded reference. They should not be silently promoted into the only evaluation set, because then the generator would help define the answer it is later credited for matching.