What an Evaluation Can See—and What Deployment Adds
This chapter examines the evidence choices inside the NIST TEVV-Athlon Framework: what controlled model tests and benchmarks establish, what red teaming, user testing, and field testing add, and why the measurement process itself must be questioned. The practical judgment is not to choose a winner among methods. It is to match each method to the risk and decision it can illuminate, then weaken a deployment claim when important parts of the intended use remain unobserved.
A benchmark can answer a narrow question well
NIST describes model testing as evaluation in a controlled computational setting. An evaluator may use held-out test data, predefined prompts, expected criteria, and statistical or signal-processing metrics to examine a model, its software processes, or its outputs. Benchmarking is a more standardized form of model testing: datasets, prompts, or tasks are used to compare outputs against expected answers and calculate metrics across systems.
The controlled setting is a strength, not a defect. It makes repeated trials possible. It can hold conditions steady while a team compares model versions, tests a known capability, or checks whether a change improved a defined metric. Automation can reduce the cost of running the same task at scale. When a model must meet a specific technical requirement, a controlled test may provide the clearest evidence available about that requirement. A deployment decision that ignores such evidence would discard useful information.
The boundary is equally important. A test can establish how the system performed on the data, prompts, labels, and scoring rules that were selected. It does not automatically establish how people will use the system, how tasks will change, how adversaries will probe it, or how the surrounding workflow will alter its outputs. A benchmark may measure a proxy—a convenient observable that stands in for a harder real-world outcome—rather than the outcome itself. Accuracy on a set of questions, for example, may be relevant to a capability while remaining silent about whether users receive suitable help in an operational process.
Public benchmarks introduce further hazards. Because many are openly available, model developers may encounter the test material during training or system development. That is data contamination: the supposedly unseen evaluation material is no longer independent of the model-building process. A benchmark can also become saturated, so many systems cluster near its ceiling, or become a target for narrow optimization. A stable score under those conditions may reflect familiarity with the test or adaptation to its scoring procedure rather than a broad improvement in the intended capability.
NIST calls the last problem a form of Goodhart’s Law: once a measurement becomes a target for optimization, it can begin to reflect the optimization process rather than the concept it was meant to measure. The effect need not involve fraud. A team that improves a benchmark score may make a real improvement on that benchmark while inadvertently changing retrieval behavior, calibration, interoperability, or downstream outcomes. A model can therefore become better at the measured task and less suitable for the larger workflow at the same time.
This is why a high score should be read with its test boundary attached. The relevant questions are: What population and task does the benchmark represent? Was the data independent of model development? What does the metric leave out? Has the system been tuned to the test? What decision is the result being used to support? NIST suggests that in-house or application-specific benchmarks can reduce some leakage and contamination risks by tailoring tests to an organization’s deployment setting. That does not make a custom benchmark automatically valid; it makes the intended context easier to represent and the remaining assumptions easier to inspect.
A controlled test is thus strongest when the claim is controlled too. “This version achieved this result on these predefined tasks” is a useful statement. “This system is ready for broad use” is a larger statement that requires evidence about more than the predefined tasks. The gap between the two is where complementary evaluation methods earn their place.
Deployment adds people, adversaries, and time
NIST groups red teaming, user testing, and field testing as ways to examine performance in more realistic settings. They do not all simulate the same reality. Red teaming intentionally probes for harmful outputs, edge cases, or vulnerabilities and asks how the system breaks under pressure. User testing examines how people understand and interact with the system, including whether its behavior aligns with user intent. Field testing observes the system in an operational environment, where real inputs, constraints, and consequences can change performance.
The differences are easier to see in the query-violation example. A chatbot may be asked to provide a meal plan, travel information, or a television recommendation while avoiding a specified undesirable response. Ordinary user testing can reveal whether people receive information they find helpful and whether they understand the answer. Red teaming can deliberately search for prompts that make the chatbot produce the prohibited response. A field activity can reveal what happens when the chatbot is embedded in a real support process, where users have different levels of expertise, time pressure, and reasons to rely on the output. The same system is present in each case, but the Event exposes a different behavior.
These methods also carry different costs and failure surfaces. Red-team results depend on the scope and creativity of the probes, and a limited exercise cannot establish that no other vulnerability exists. User testing depends on who participates, what instructions they receive, and how their experience is recorded. Field testing can be resource-intensive, difficult to reproduce, and affected by conditions that the evaluator cannot fully control. Studies involving people may require informed consent, data protection, and legal or ethical oversight. A realistic method can therefore be more relevant to a deployment decision without being more precise in every respect.
The four evidence paths should be read as complements, not as a ladder in which one permanently supersedes another. Controlled testing provides comparability and repeatability. Red teaming explores deliberate misuse and edge cases. User testing reveals interaction and interpretation. Field testing shows how the system behaves within the conditions that matter to operations. The decision layer then asks whether the combination is sufficient for the stated use. If a system is high-risk, the case for broader coverage and greater rigor becomes stronger; if the decision is limited to a narrow build-stage question, a smaller assessment may be appropriate.
The right comparison is therefore not “Which method is best?” but “Which relevant condition does this method expose, and which condition remains hidden?” No single method dominates across all goals, risks, and constraints. A benchmark may be the best way to compare two model versions on a defined capability. A red-team exercise may be the best way to discover whether safeguards fail under adversarial prompting. Neither can replace the other when both capabilities and misuse matter. A field pilot may provide necessary context for an operational decision, but it cannot make a poorly defined metric meaningful.
The instrument must survive interrogation
NIST extends the question of validity from the AI system to the instruments used to evaluate it. Prompts, datasets, benchmarks, survey questions, interview protocols, and scoring rubrics can all be useful or defective depending on how they are designed and applied. A benchmark developed for one setting may offer limited insight in another. A survey that asks about convenience may fail to detect harmful reliance. An annotation scheme that leaves a key category ambiguous can turn disagreement among reviewers into apparent variation in the model.
This is a construct problem. A construct is the concept the evaluation intends to represent—such as helpfulness, safety, fairness, or reliability. The observable measure is only an operational definition of that concept. The evaluator must ask whether the observations cover the important parts of the construct, whether the scoring rule rewards the behavior the organization actually wants, and whether the result relates to outcomes outside the test. More decimal places do not repair a weak connection between a metric and its construct.
Scientific measurement practices make that interrogation more disciplined. NIST recommends clearly defining what is measured and how the result will be interpreted. It points to separating training and test data to reduce contamination, documenting assumptions and limitations, reporting variability and uncertainty, and comparing performance with appropriate baselines. Structured experimental designs should identify relevant variables and conditions, use testable hypotheses where appropriate, and repeat trials when stochastic model behavior or sampling variability could change the result. Calibration against ground truth or known outcomes can improve consistency over time.
Each practice answers a different source of doubt. A baseline tells the organization whether a new model is better than a prior version, reference system, or alternative method; an absolute score alone may not. Repetition reveals whether a result is stable or happened under one random sample or one model run. Recording variables and conditions makes it possible to ask whether a change in users, prompts, data, or surrounding software could explain a change in outcomes. Uncertainty reporting shows how much confidence belongs around an estimate rather than presenting a sample result as an exact property of the whole population.
Limitations are part of the result, not an apology appended to it. If an evaluation included only a narrow user group, the claim should retain that boundary. If field conditions were difficult to reproduce, later readers should know which conditions may have influenced the outcome. If the metric was chosen because it was feasible rather than because it captured the whole risk, the organization should say what remains unmeasured. This information helps a decision maker choose among a limited deployment, another test, a monitoring requirement, or a refusal to generalize.
Measurement validation is also ongoing. NIST emphasizes that different deployment settings, user populations, operational environments, and changing system capabilities can alter the effectiveness of a TEVV-Athlon. A test that was suitable for a prototype may not be suitable after the system is integrated into a larger workflow. A benchmark that once distinguished systems may lose information as developers optimize against it. A rubric that worked for one class of user may fail when the population changes. The question “does this measure what we intend?” must therefore be revisited when the object, context, or decision changes.
The later cryptographic case makes Goodhart’s warning more adversarial: it examines a provider that can shape training data around known audit evidence, producing a stronger local result without the intended behavior elsewhere. That case is not evidence that every benchmark optimization is malicious. It is a reminder that an evaluator should consider not only accidental mismatch but also who knows the measurement, who can adapt to it, and what incentives the result creates.
A defensible deployment claim consequently has a layered form. It names the task and population, identifies the property and its operational measure, states which conditions were tested, combines evidence suited to ordinary use and failure behavior, reports uncertainty and limitations, and identifies what will be monitored later. The claim may still be conditional, but its conditions are visible. The next step is to bring this model together: before asking what cryptographic certification can add, a reader should be able to inspect whether the goal, target, evidence, sample, context, and decision already align.