What Trustworthy AI Grading Can Actually Claim
The central question is no longer whether an AI grader can agree with a teacher. The evidence shows that it often can, under defined course conditions. The harder question is what that agreement warrants us to say about the system. This final synthesis brings together the two real-course studies to establish a claim ladder: operational alignment, faithful input interpretation, correct judgment, useful feedback, human variation, and institutional readiness. The conclusion is not that AI grading has no value. It is that trustworthy claims must be narrower than the number used to advertise them.
The claim ladder behind a grade
Teacher or TA agreement belongs at the first rung because it answers a real institutional question. If an assistant is meant to support an existing course, it must be compatible with the course’s grading practice. The UC Irvine study provides substantial evidence for that limited claim. Across 3,945 handwritten free-response records, the AI–TA score gaps centered near zero. The AI scores averaged 0.40 points below the TAs, with a standard deviation of 1.12. Across quizzes, mean absolute gaps generally ranged from 0.5 to 1.06 points, and 68% to 86% of scores fell within one point of the TA score.
That is more than a demonstration on a handful of carefully prepared answers. It is evidence from authentic, handwritten, course-scale work. It suggests that an AI assistant can reproduce the approximate outcomes of current grading practice often enough to deserve serious investigation as a support tool. It does not turn the TA score into a universal truth label. The comparison is bounded by the included records, the course’s rubrics, the problem types, and the fact that TA grades remained the official outcomes.
The second study reaches a similar conclusion through model comparison. Across two open-ended data analytics and machine learning practices, DeepSeek-R1 was the only model described as consistently aligned with the manual grades. Its mean absolute error was 19.17 points for the first practice and 19.01 for the second, and the reported tests did not find statistically significant differences from the instructors under the study’s criteria. That makes it the closest match to the human benchmark in that experiment.
The wording matters. “Closest match in this course under this prompt” is a defensible result. “Most accurate AI grader” is a larger claim than the evidence can support. The model comparison involved 110 students’ deliverables, one course, two practices, one set of instructor judgments, and a defined evaluation procedure. A model can match a local reference because it has been well aligned to the local rubric, because it shares the instructors’ strictness or generosity, or because both overlook the same kind of problem. Alignment is necessary for practical adoption, but it is not the final rung.
The next rung is input fidelity. Before asking whether a score is right, an evaluator must ask whether the system saw the student’s work accurately. This is obvious in the handwritten-calculus setting, where segmentation and transcription stand between the page and the grading model. It is less obvious in a notebook, where the evaluator must still verify that code, outputs, explanations, and relevant context were available in the form the model actually processed.
Input fidelity is not a minor technical footnote. If a negative sign disappears, a fraction is mis-scoped, a crossed-out attempt is treated as final, or a blank area is filled with invented content, the later score is based on a changed response. A close score can then be accidental. A wrong input can also be masked by a coarse rubric, producing the same total that a human assigned for a different reason. Agreement after an unverified transformation is weaker evidence than agreement after the representation has been checked.
The third rung is judgment correctness. Here the question is not whether the final number resembles a teacher’s number, but whether the evaluator recognized the right mathematical or methodological facts. Independent reviewers add evidence at this level. In the calculus study, they examined input quality, score alignment under the stated rubric, rubric fidelity, and feedback accuracy. Their review found fully correct, acceptable, and incorrect evaluations rather than treating every near-match as equivalent. It also exposed two distinct errors: over-penalizing valid methods that the rubric did not list, and being too lenient toward familiar-looking explanations whose reasoning was wrong.
Human variation belongs on this rung as well.