What Counts as Evidence When a Test Goes Well?

This chapter examines how to interpret medical-device validation results when users appear to succeed. The practical question is what each kind of evidence can establish: observed performance, knowledge-task answers, close calls, repeated attempts, confusion, and post-use interviews. A credible judgment does not compress those signals into one satisfaction score. It combines them to determine whether a critical task was completed, what the user understood, why a problem occurred, and whether the remaining risk requires a design change.

A test can look reassuring and still contain information that matters more than the headline result. Users may finish a task after a fragile recovery. They may misunderstand a warning without being asked about it. They may make an error and not notice. They may report that nothing was difficult even though the observer saw a risky action. The FDA’s approach treats these differences as reasons to examine the evidence more carefully, not as contradictions to be averaged away.

Different data answer different questions

Observed performance is the first layer because some safety questions are about what the user actually did. If a critical task requires a sequence of actions, an observer can determine whether the participant completed the sequence and achieved the intended outcome. This is stronger than relying only on a participant’s opinion about whether the task went well. People may be confident about an action they performed incorrectly, or they may fail to notice that the device entered an unexpected state.

Observation does not mean that every movement must be timed or scored as a defect. The FDA says task timing is meaningful in validation when speed is clinically relevant, such as with an emergency device. Timing a task that has not been defined in advance as time-critical can distract from the more important question of whether the user performed it safely and understood what happened. A slow but correct action and a fast but incorrect action do not have the same safety meaning.

Knowledge tasks answer a different question. Some critical information cannot be evaluated simply by watching the device being used. A user may need to understand a contraindication, a warning, an environmental limit, a maintenance requirement, or the consequence of reusing a component. The person may never encounter the relevant condition during the simulated workflow. Asking an open-ended, neutral question can reveal whether the information became usable knowledge rather than merely appearing in a manual or on a label.

This distinction matters because visible behavior can conceal misunderstanding. A participant may perform a routine sequence correctly while not knowing what to do if the environment changes or an important warning appears. Conversely, a participant may answer a knowledge question imperfectly while still using the device safely in the tested scenario. The two findings should not be treated as interchangeable. One concerns action in a use scenario; the other concerns understanding needed for safe decisions.

Close calls add a third kind of evidence.