Validation Needs More Than One Witness

An AI-grading system makes several different claims at once, so no single witness can validate all of them. This chapter turns the cases and pipeline analysis into a practical evidence stack: teacher or TA alignment for operational fit, independent review for defensibility, input checks for faithful reading, student response for usability, and qualitative audits for contextual reasoning. The aim is not to average these signals into one impressive number. It is to make each signal answer the question it is actually capable of answering.

Give each witness a different job

The teacher or TA is the first witness because the system is entering an existing educational institution. A course already has expectations about partial credit, acceptable methods, explanation, and the relative importance of errors. Comparing AI scores with official human grades tests whether the system can operate within that local practice. The UC Irvine study’s 3,945-record comparison is valuable for precisely this reason: the AI–TA gaps were concentrated near zero, the average AI score was 0.40 points lower, and quiz-level mean absolute gaps generally ranged from 0.5 to 1.06 points.

Those results would be difficult to replace with a purely abstract test. A model might solve textbook problems well and still assign scores that are unusable in a particular course. Alignment is therefore a necessary first layer. But it is a layer about compatibility with an operational reference, not a complete judgment about correctness.

The second witness is an independent reviewer. In the calculus study, reviewers examined 3,851 handwritten free-response records for input quality, score alignment under the study rubric, rubric fidelity, and feedback accuracy. More than 90% of AI–reviewer score differences fell within one point. Reviewers classified 79.79% of the evaluations as fully correct, 9.55% as acceptable, and 10.67% as incorrect.

This adds information that a TA comparison cannot supply by itself. It can reveal an OCR problem that happened not to change the total, a mathematically valid method that the rubric failed to recognize, or a feedback explanation that sounds coherent but does not follow the student’s reasoning. Because the reviewers are not simply the original course graders, their judgments reduce circularity.

Independent review is not a magic escape from human assumptions. The reviewers used the same study rubric, and the study itself reports meaningful human disagreement. The review can be more decoupled than a direct AI–TA comparison without becoming universal ground truth. Its proper job is to test whether the AI output remains defensible under a stated standard and to identify failure patterns that aggregate alignment misses.

The third witness is human variation itself. The calculus study compared two independent reviewers using the same criterion-based rubric on 171 question instances. Exact agreement occurred for about 45% of the questions, with nontrivial differences across several score levels. This is not a nuisance statistic to hide. It tells us how much uncertainty is present in the reference process before AI performance is judged.

Human variation helps interpret a gap, but it cannot excuse every gap. A small difference on a borderline or unconventional solution may be consistent with ordinary judgment variation. A large difference on a core mathematical error may still be a serious model failure even if reviewers sometimes disagree. The comparison should therefore classify the reason and consequence of disagreement, not merely ask whether the AI fell inside a broad human range.