One Grade, Several Different Claims
This chapter examines what an AI-grading result actually tells us when it is compared with a teacher’s score. The immediate goal is practical: to separate evidence of operational alignment from evidence of faithful reading, correct judgment, useful feedback, and readiness for deployment. Once those claims are kept apart, statistics such as agreement rate and mean absolute error become more informative—not less—because we can see the question each one answers.
A score gap answers a narrow question
The clearest result from the UC Irvine calculus study is a score-alignment result. For each handwritten free-response question, the researchers recorded an AI score, a TA score, and the difference between them. Across 3,945 records, the gap distribution was concentrated near zero. The AI scores averaged 0.40 points lower than the TA scores, with a standard deviation of 1.12. At the quiz level, mean absolute gaps generally fell between 0.5 and 1.06 points, while 68% to 86% of scores were within one point of the TA score.
That evidence supports a bounded conclusion: under the tested course conditions, the AI grader often produced scores close to the scores that TAs assigned. It does not support the stronger conclusion that the AI was correct at the same rate, or that the feedback was as good, or that the system would behave similarly in another course. The comparison is against a practical reference, not against an independent inventory of mathematically correct judgments.
Mean absolute error is useful because it summarizes the size of score differences without allowing positive and negative gaps to cancel each other out. A model that is five points too high on one response and five points too low on another has a mean signed error of zero, but its mean absolute error is still five. Yet even MAE remains an outcome-alignment statistic. It does not reveal whether the error came from a misread symbol, an overly rigid rubric, a missed methodological flaw, or a defensible difference between two human graders.
The “within one point” rate has a similar strength and limitation. It is easy to interpret as a tolerance band, but it depends on the scoring scale, the distribution of cases, and the chosen reference. A one-point difference may be inconsequential on a large quiz item and serious when it changes whether a student receives credit for a critical step. Aggregate agreement can also coexist with a small number of severe errors. The tails matter because a grading system is often used precisely where a student’s unusual solution or an ambiguous case needs judgment.
The second study makes reference-dependence visible through model comparison. It evaluated six language models on two open-ended data analytics and machine learning practices. DeepSeek-R1 was described as the only model that consistently aligned with manual grading across both practices. Its mean absolute error was 19.17 points for the first practice and 19.01 for the second, compared with manual mean grades of 54.53 and 67.16. The reported paired tests did not find statistically significant differences between DeepSeek-R1 and the instructors under the study’s criteria.
This is a meaningful result for the question the study asked: which tested model most closely reproduced these instructors’ grade distribution for these assignments and this prompt? It is not a proof that DeepSeek-R1 is the most accurate grader in general. The result comes from 110 students’ deliverables in one course, two practices, one structured prompt, and one set of instructor judgments. A model can be the best match to a local benchmark because it shares the benchmark’s preferred strictness, generosity, or blind spots. A different course could produce a different ordering without either study being internally wrong.
The same caution applies to statistical significance. A non-significant difference says that the study did not detect a reliable difference from the chosen human reference at its specified threshold. It does not show that every individual grade is correct, that no systematic error exists, or that the model’s comments are pedagogically adequate. A significance test is a statement about a comparison under a design; it is not a certificate attached to the model.
The denominator also carries meaning. The calculus comparisons excluded some work because of segmentation or scan problems, failed links to TA records, incomplete exports, or review-capacity constraints. The paper reports these exclusions as driven by processing and logistics rather than grading outcomes, but they still define the population represented by the statistic. A result about successfully processed, linkable responses is not automatically a result about every response a course receives. Validation should therefore report not only how close the included scores were, but also which cases could not enter the comparison and why.
The shape of the errors matters as well. An average can look reassuring when routine responses dominate and unusual solutions, faint writing, or methodologically difficult assignments carry the largest risks. The calculus study’s score-gap tails and the data-analytics study’s large qualitative divergences point to the same practical lesson: the cases most likely to require judgment are often the cases least well summarized by an average. A validation claim becomes more useful when it identifies where performance changes, not only where it is typical.
The objects hidden behind a final grade
A final grade compresses several judgments into one visible output. To validate the system, those judgments need to be inspected separately.
The first object is the student’s original work. In the calculus pipeline, the system segmented answer regions, transcribed handwriting into LaTeX, and passed the transcription to a rubric-guided language model. Roughly 88% of transcriptions were considered acceptable on the course data. On a fixed challenge subset of 171 solutions, GPT-4.1-mini reached 84% acceptable transcription compared with 55% for Mathpix. The errors in the remaining cases included ambiguous nested fractions, diagrams, crossed-out work, blank-answer hallucinations, and occasional autocorrection of a student’s mistake.
Those numbers are not merely engineering diagnostics. They define the input on which the grading judgment rests. If a fraction bar is assigned the wrong scope, a negative sign disappears, or an erased line is treated as the final method, a later score comparison becomes difficult to interpret. A matching TA score might mean that the transcription error had no effect on the total. It might also mean that both systems happened to award the same coarse score for different reasons. Without checking the input, score agreement cannot establish input fidelity.
The second object is the grading rule. The calculus study used a flexible rubric to credit diverse valid approaches and a fixed rubric to check required components and common errors. The system evaluated both and selected the higher score. That max-rule produced the lowest mean absolute error against TAs on the tested quizzes, but the rule’s success is local to those questions and rubric choices. It tells us that a particular aggregation of two grading perspectives fit the reference well; it does not tell us that the aggregation is universally optimal.
The rubric itself also determines what a score means. A coarse human rubric may place many forms of incomplete understanding into one low-credit bin, while a fine-grained AI rubric may award points for several isolated components. The AI score can then be higher without the AI having discovered a hidden truth; the two graders are measuring different resolutions of the same work. Conversely, a fixed rubric may withhold credit from a correct noncanonical solution while a flexible rubric recognizes it. A score comparison without rubric comparison leaves the construct partly hidden.
The third object is the explanation attached to the grade. The calculus study’s independent reviewers examined not only score differences but also rubric fidelity and feedback accuracy. Across 3,851 reviewed responses, 79.79% of evaluations were judged fully correct, 9.55% acceptable, and 10.67% incorrect. Reviewers found two important patterns beyond OCR failures: the model sometimes over-penalized valid methods missing from the rubric, and it sometimes rewarded explanations that resembled familiar rubric steps even though the reasoning was wrong.
The data-analytics study found a related gap through a different evaluation design. DeepSeek-R1 produced the highest aggregate similarity score across the study’s lexical, semantic, and BERT-based measures in both practices. But the study also found that model feedback was much longer than instructor feedback, and it warns that semantic similarity and verbosity do not automatically equal pedagogical quality. In a stratified qualitative review of 243 cases, at least one instructor judged 194 cases to be informative but lacking contextual depth. Reviewers identified minor but important mistakes missed in 148 cases and subtle methodological arguments missed in 189.
These observations explain why a score metric and a feedback metric should not be collapsed into one overall quality number. The metrics observe different outputs. A score comparison asks whether two evaluators assigned similar numerical outcomes. A semantic metric asks whether two texts resemble each other in language or meaning. Neither one, by itself, asks whether the feedback identified the decisive misconception, explained its consequence, and gave the student a useful next step.
A fourth object is the use decision. Even an accurate score and a good explanation may not justify autonomous grading in every context. The calculus evidence comes mainly from low-stakes quizzes. The data-analytics study describes the model outputs as instructor-reviewed drafts. The cost, privacy, and review burden of a system also matter, but they are not hidden dimensions of score accuracy. They are separate conditions for responsible use.
Read every result as a bounded claim
A useful discipline is to attach four pieces of information to every validation result: the reference, the metric, the task, and the boundary. The reference identifies whose or what judgment is being compared. The metric identifies whether the result concerns score distance, transcription quality, feedback resemblance, or something else. The task identifies the course, assignment, and type of student work. The boundary states what the result leaves unresolved.
| Validation result | What it can support | What it cannot establish alone |
|---|---|---|
| AI score close to TA score | Compatibility with current course grading practice | Absolute correctness, faithful reading, or useful feedback |
| Low mean absolute error | Small average numerical deviation from the reference | Correct rationale, tail-risk control, or transfer to another course |
| Acceptable OCR or input rate | Quality of the representation supplied to the grader | Correct scoring on every legible response |
| Similarity to instructor feedback | Resemblance in content or language | Contextual diagnosis, actionability, or learning impact |
| Positive student perception | Clarity, acceptability, or trust in the output | Mathematical or methodological truth |
| Independent reviewer agreement | A partially decoupled check under a stated rubric | Universal ground truth or independence from all human assumptions |
The table is not a ranking of evidence sources. It is a reminder that a result answers the question built into its comparison. A high within-one score can coexist with unacceptable OCR on a smaller class of difficult submissions. High semantic similarity can coexist with generic advice. Positive student reactions can coexist with a subtle mathematical error. None of these possibilities makes the original metric useless; each shows why its interpretation must remain bounded.
The practical implication is not that institutions should ignore teacher agreement. They should report it clearly, because a system that does not fit current grading practice may create immediate confusion and inequity. But they should label it accurately: it is evidence of operational alignment. To make a broader validity claim, the evaluation must add evidence about what the system saw, how its rubric handled valid variation, whether its explanations were correct and context-sensitive, how it behaves on difficult cases, and what happens when uncertainty is detected.
That is also why the next question is about the human reference itself. If teacher and TA scores are indispensable for understanding whether a system fits a course, but no single score is guaranteed to be the only defensible one, then validation must characterize the reference rather than silently treat it as an oracle.