The TEVV-Athlon: From Organizational Goals to Evidence
This chapter focuses on the operating model inside NIST’s TEVV-Athlon Framework for Evaluating AI Systems: how an organization moves from a concern about an AI system to evidence that can inform a decision. The useful judgment is not whether the framework has four memorable labels, but whether each stage preserves the meaning of the original question. If a goal is vague, a metric can become a convenient substitute; if a metric is precise but poorly connected to the goal, a perfectly executed test can still support the wrong conclusion.
First, decide what the organization needs to know
NIST presents the TEVV-Athlon Framework as a way to create a customized assessment, not as a universal scorecard. The assessment can examine system attributes, trustworthiness characteristics, and operational goals that matter to a particular organization. That starting point is important because “quality” is not one property of an AI system. Accuracy, safety, privacy, fairness, reliability, and usefulness may overlap in a real application, but they do not ask the same question or require the same evidence.
The framework’s first stage, Articulate & Organize, is where the organization decides what the assessment is for. The goal should be stated plainly enough that someone outside the specialist team can understand it. The organizers should also identify who will care about the result and how it could change a decision. Time, cost, resources, existing methods, conditions for success, and likely challenges belong in the same conversation. These are not project-management details added after the “real” technical work. They determine what can be measured, for whom, and with what consequences.
Consider an organization evaluating a chatbot used in a customer-support process. “Determine whether the chatbot is trustworthy” is too broad to control the assessment. A more useful question might ask whether it can provide relevant answers in defined support scenarios without producing a specified class of harmful response. The second question identifies an intended use, a class of behavior, and a reason the result matters. It also makes room for a practical constraint: an organization may have the resources to run automated prompt tests but not to recruit a large group of users or observe a long field deployment. That constraint should shape the design openly rather than silently narrow the claim after the results arrive.
Lifecycle stage matters for the same reason. NIST describes stages such as planning and design, building and using, deploying, and operating and monitoring. A test that is feasible while a model is being built may be different from an evaluation after people interact with the system in its operational environment. The intended use of the result changes too. Early evidence may guide whether to continue development; deployment evidence may inform procurement, release, escalation, or monitoring. The model has not necessarily changed, but the question surrounding it has.
The first stage is also iterative rather than a one-time form to complete. NIST notes that some of the questions are difficult to answer at the outset and that answers can become more definite as the design progresses. That does not make the initial goal optional. It means the organization should record what is known, what is estimated, and what must be learned before the assessment can support a consequential decision. If the available resources cannot support a proposed Event, the honest choices are to change the Event, narrow the intended claim, or defer the decision—not to retain the broad claim and quietly accept thinner evidence.
Once the goal and context are clear, Define & Construct turns the broad concern into a measurement target. NIST calls these targets Metrology Blocks, or simply Blocks. A Block is the particular concept or metric the assessment will evaluate, chosen in relation to the relevant system attribute or trustworthiness characteristic. It must have a precise definition of the evidence required. That definition is the interface between an organizational concern and an observable result. It also sets the evidence boundary. A Block about helpfulness might require reports from users who attempted a defined task; a Block about violation frequency might require a rule for classifying outputs across ordinary and deliberately difficult prompts. The definition does not yet tell the evaluator whether the result will generalize, but it makes clear what kind of observation must be gathered before that question can even be asked.
The interface can fail in two opposite ways. If the Block is merely “overall trust,” it gives an evaluator no stable target and allows different people to interpret a pass differently. If it is an easy-to-compute number that has little relationship to the concern, precision only makes the mismatch harder to notice. A Block should therefore answer two questions at once: what aspect of the system is being examined, and what observation would count as evidence about that aspect? The framework allows more than one Block for a characteristic, and different assessments can choose different Blocks for the same characteristic because the operational context may differ.
The diagram shows two kinds of connection. The stages form a sequence because the output of one constrains the next: there is no well-defined Event until the Block is clear, and no meaningful synthesis until the evidence can be related back to the goal. The AI Risk Management Framework adds an organizational loop around that sequence. Govern and Map provide governance and operating-environment information; the TEVV-Athlon performs the Measure function; its results inform Manage. The assessment therefore sits inside a decision system rather than above it.
NIST’s query-violation example shows the narrowing process. The evaluation concerns chatbots that should answer queries without giving away prohibited information. The relevant characteristics are “Valid and Reliable” for the desired response and “Safe” for avoiding violations. The broad characteristics become two more specific Blocks: Helpfulness, measured from users’ perspectives, and Violation Frequency, measuring how often prohibited responses occur. The Blocks are not yet a test procedure. They are the targets that the rest of the assessment must serve.
That separation is the first protection against an inflated claim. An organization that measures helpfulness cannot report the result as a complete safety finding unless it has also defined and measured the safety concern. Likewise, a low violation frequency in one scenario does not silently become a guarantee about every query, user, or operational setting. The target must remain attached to the boundary within which it was defined.
Then build evidence and return it to the decision
The third stage, Apply & Measure, specifies how evidence for each Block will be produced. NIST calls those activities Events. An Event is not the property being measured; it is an activity selected because it can produce information about that property. The methods and instruments used to elicit, collect, and analyze that information form the Toolbox. This may include instructions, prompts, questionnaires, annotation rules, benchmark tasks, red-team methods, or statistical procedures, depending on the assessment.
This distinction gives the framework an operating model. The organization owns the question and the decision context. The assessment design turns that question into Blocks. Events create situations in which the relevant behavior can be observed. Tools control how participants or evaluators act, how outputs are recorded, and how observations are classified. The resulting data is not automatically an answer; it is material that must be interpreted against the Block’s definition. Each boundary matters because a failure at one interface can contaminate the next stage. A questionnaire that asks only whether a response felt convenient may not capture harmful misinformation. An annotation schema that does not define a violation consistently can make the resulting frequency look more exact than it is.
The boundaries also determine where a later dispute belongs. If stakeholders disagree about what “safe” means, the problem is in the goal or Block, not in the arithmetic used to summarize responses. If the Block is sound but an Event never exposes the relevant behavior, the problem is evidence coverage. If the Event produces useful observations but the Tool records them inconsistently, the problem is measurement quality. If all of those pieces work and the result is still applied to a different population or lifecycle stage, the problem is claim scope. Separating these failure locations helps an organization repair the right layer instead of adding confidence language to an unchanged design.
In the query-violation assessment, User Testing is an Event in which people interact with chatbots to obtain information relevant to their queries. User-testing instructions elicit the task, and a post-task questionnaire collects information about the users’ experiences. That path supplies evidence for Helpfulness. Red Teaming is another Event: participants deliberately prompt the chatbots in attempts to make them reveal violations. Red-teaming instructions guide those attempts, and an annotation schema helps identify whether a conversation contains the prohibited behavior. User Testing can also contribute evidence to Violation Frequency, because violations may occur during ordinary use; Red Teaming adds pressure that ordinary users may not generate.
The important point is not that one Event is always better. It is that each Event exposes a different part of the system. User testing can reveal whether people understand and can use an answer. Red teaming can expose failures under deliberate probing. A benchmark can make a controlled comparison repeatable. A field activity can show how behavior changes in an operational environment. The Toolbox is the combination required by the Blocks and the decision, not a catalogue of every available method.
The final stage, Synthesize & Interrogate, is where collected observations become an answer that an organization can use. NIST describes this as reviewing the stages in reverse order. The analysis begins with what the Toolbox produced and what happened during each Event. It then relates those observations to the corresponding Blocks, asks whether they answer the original assessment goals, and reports the implications for the AI system. In the example, descriptive statistics can summarize questionnaire responses for Helpfulness and violation frequency across User Testing and Red Teaming. Those summaries provide evidence about validity and safety in the defined query-violation scenarios; they do not erase the scenarios’ boundaries.
A decision maker then has to connect the summaries to an action without treating the analysis as a second, hidden definition of success. If the organization cares about harmful responses in a particular support workflow, it must decide how much evidence is enough for that workflow, which failures require escalation, and whether the observed users and prompts resemble the intended population. A descriptive result can inform that judgment, but it cannot supply missing coverage or decide the organization’s risk tolerance. Keeping the original goal visible is what prevents synthesis from becoming a new score detached from the assessment that produced it.
“Interrogate” is as important as “synthesize.” The evaluator must ask what the data means, what it does not mean, and whether the measurement remains connected to the intended concept. A result can be internally consistent and still be too narrow for the decision. A high average can conceal a serious tail of failures. A low violation rate can reflect limited prompting rather than robust safety. If the organization intends to deploy broadly, the result may need additional evidence from more realistic settings, more relevant users, or a later lifecycle stage. NIST’s framework does not prescribe one automatic escalation rule; it makes the need for such judgment explicit.
This is why completing the four stages is not the same as earning an unconditional deployment guarantee. The stages provide a disciplined route from goals to evidence and back to decisions. They do not make an unsuitable Block suitable, a biased sample representative, or a controlled Event equivalent to real use. Their value lies in keeping those questions visible and in making it possible to locate the point where a claim becomes stronger than its evidence.
A good TEVV-Athlon thus ends with a decision that names its conditions. It might support further development, a limited deployment, additional testing, procurement, or continued monitoring. It might also show that the current evidence is insufficient. That is not a failure of the framework; it is one of the outcomes a decision-oriented assessment should be able to produce. The next question is what kinds of evidence can fill the gap between a controlled assessment and the world in which the system will operate—especially when benchmarks, users, adversaries, and environments expose different behaviors.