Testing the System You Actually Deploy

A public-sector AI system can pass an impressive demonstration and still be unreliable in the workflow where an agency uses it. The important question is therefore not only whether the vendor met an award-time performance claim, but whether the agency can keep producing trustworthy evidence as models, data, users, and operating conditions change. This is what continuing evaluation is for. It separates acceptance of a capability from assurance that the capability remains fit for a public task.

From a benchmark to a live service

A benchmark answers a narrow question under defined conditions. Acceptance testing asks whether a proposed system meets the requirement for which the agency is considering deployment. Operational monitoring asks a harder question: what is the system doing now, for these users, with this data, inside this workflow, and with what consequences? The first two forms of evidence remain valuable. They establish a baseline and can prevent an agency from committing to a capability that fails before it starts. They are not substitutes for the third.

GAO reports that officials at all five selected agencies identified early testing and continuous evaluation as challenges. NIST officials told GAO that universal AI tests did not yet exist because systems and services vary in capability and complexity. That variation makes a generic score difficult to interpret. A model can perform well on a public benchmark but fail on the agency’s data, or perform acceptably on an internal test while producing outputs that do not improve the public outcome. The agency has to define the purpose of the test in relation to the mission rather than treating any available number as a complete answer.

This is also why the agency needs an evidentiary position of its own. OMB guidance summarized by GAO encourages agencies to test proposed solutions before award and to monitor and evaluate them across the acquisition lifecycle. For independent evaluations, the guidance calls for validation and testing data defined by the agency and not accessible to the vendor, while keeping those data as similar as possible to the data used in deployment. The point is not to create an adversarial ritual. It is to prevent the supplier from being the only party able to choose the examples, define the conditions, and report the result.

An agency-defined validation set is not automatically representative or fair. It still needs technical and domain review. But it gives the agency a reference against which a vendor’s claim can be examined. It can reveal whether the system handles the cases that matter locally, whether errors concentrate in a subgroup, whether a confident output is often wrong, or whether a new version changes behavior in a way that matters to the mission. The evidence can then support a decision to continue, constrain, remediate, or stop—not simply a declaration that the system has been tested.

GAO’s example of GSA’s USAi illustrates the continuing nature of that comparison.