Why Teacher Agreement Is Not Enough
AI-assisted grading is not simply a language model producing a number. It is an assessment system that takes student work, interprets it, applies a rubric, assigns a score, and often explains that score in feedback. This series examines what it means to validate such a system in real educational settings. Its central question is narrower—and more important—than whether an AI score resembles a teacher’s score: what, exactly, does that resemblance prove? The answer helps readers distinguish a useful grading assistant from a system that only looks reliable when judged by one convenient comparison.
The attraction of agreement at scale
The appeal of AI grading begins with a practical problem. Large courses generate more written work than instructors and teaching assistants can assess with equal speed, consistency, and detail. When feedback arrives late or consists of little more than a score, students lose the chance to use it in the next stage of learning. An automated assistant promises to relieve repetitive work while returning explanations quickly and consistently.
The two studies examined for this series make that promise concrete in different ways. One followed an OCR-plus-LLM system deployed across a large single-variable calculus sequence at UC Irvine. The system processed handwritten quiz solutions: work that contains incomplete steps, crossed-out attempts, faint symbols, unusual notation, and partial credit. Across three academic terms, the deployment covered 20 quizzes and more than 1,000 students. The detailed comparisons focused on free-response work from the 2025 spring term, while teaching-assistant scores remained the official course grades. AI results were supplied for reference rather than used to replace the course’s existing grading authority.
The other study examined 110 students’ Python notebooks in a university course on data analytics and machine learning. These were open-ended assignments, not answers with one predetermined route. Students could choose methods and were expected to explain their analysis. Six language models produced grades and written evaluations that were compared with instructor assessments. The study proposed a framework that looks at technical performance, pedagogical value, and institutional constraints rather than treating model selection as a contest with one universal winner.
These settings differ, but they create the same validation problem. In both, human evaluations are the most relevant practical comparison available. Teachers and TAs determine course outcomes; their rubrics embody local expectations; their feedback reflects the context of the assignment. If an automated system consistently departs from those judgments, it may be unusable even when it performs well on an abstract benchmark. Agreement therefore matters. It tells an institution whether the system can fit an existing assessment practice closely enough to be considered for assistance.
But operational fit is only one kind of validity. A system can match an official score because the score is coarse, because the rubric rewards a familiar pattern, because an upstream transcription error happens not to change the total, or because the model and the human share the same blind spot. A matching number can be an encouraging signal without being a verdict on the quality of the judgment that produced it.
A matching score is a narrow success
The calculus study offers strong evidence that AI scores can align with current TA practice. Across 3,945 handwritten free-response student-question records, the distribution of AI–TA score gaps was concentrated near zero. The AI scores averaged 0.40 points below the TA scores, with a standard deviation of 1.12. Across individual quizzes, mean absolute gaps generally ranged from 0.5 to 1.06 points, and the proportion of scores within one point ranged from 68% to 86%.
Those results answer a meaningful question: can the system behave roughly like the people who currently grade this course? For a low-stakes quiz, that is part of the case for using it as an assistant. It may also help identify where grading is unusually inconsistent or where a human should inspect a result. Yet the comparison does not establish that the AI is absolutely correct, because the TA score is the reference used to define the comparison.
The data-analytics study makes the same issue visible through model selection. DeepSeek-R1 was the only model described as consistently aligned with instructor grades across both practices. Its mean absolute error was 19.17 points for the first practice and 19.01 for the second, and its differences from the manual grades were not statistically significant under the study’s reported tests. Other models, particularly the Llama models and several models on the first practice, tended to assign higher grades.
That finding is useful, but its precise meaning is limited: DeepSeek-R1 was the closest match to these instructors, these assignments, this prompt, and this task distribution. It does not prove that the model is intrinsically the most correct grader in another course. Statistical non-significance indicates closer alignment with the chosen reference; it does not certify the absence of bias, the quality of explanations, or the correctness of every individual decision.
This distinction matters because human grading itself is not perfectly fixed. The calculus study notes that partially correct or unconventional solutions may not have one uniquely correct score. In an appendix comparison, two independent reviewers using the same criterion-based rubric agreed exactly on about 45% of 171 question instances; differences across multiple score levels were common enough to provide a meaningful human baseline. That does not make human grading arbitrary. It shows that a score is often a reasoned judgment within a rubric, not a physical measurement with one inevitable reading.
The sensible conclusion is neither “teachers are always right” nor “teacher agreement is irrelevant.” A teacher or TA score is an essential operational reference. It tells us whether an AI system fits the course’s current practice. It also supplies valuable context for interpreting unusual cases. What it does not do by itself is answer every other question an institution should care about.
What agreement leaves untested
The first unanswered question comes before grading: did the system faithfully read the student’s work? In the calculus pipeline, answer regions were segmented, handwriting was transcribed into LaTeX, and the resulting text was passed to a rubric-guided language model. The study reported that roughly 88% of transcriptions were acceptable, and that GPT-4.1-mini reached 84% acceptable transcription on a challenging 171-solution subset, compared with 55% for Mathpix. The remaining errors were not merely cosmetic. Nested fractions could be misread, diagrams were difficult, crossed-out work could be mistaken for valid work, and a sparse answer box could trigger invented content. The model sometimes silently corrected a student’s mathematical error during transcription, even though an explicit instruction reduced that behavior.
A score that agrees with a TA cannot certify an input that was never checked. A lost minus sign may change the grade; a misread final answer may make a correct solution appear wrong; a forgiving rubric may conceal the error. Input fidelity is therefore a separate validation claim.
The second unanswered question is whether the score reflects sound reasoning rather than a superficial match to the rubric. The calculus system used both a flexible rubric, intended to recognize diverse valid approaches, and a fixed rubric, intended to check required steps and common errors. It selected the higher of the two scores. This max-rule aligned best with TA scores in the tested quizzes, but the two rubrics sometimes disagreed for understandable reasons. A noncanonical solution could be mathematically sound yet lose points under a rigid checklist. Conversely, an explanation could follow familiar steps while containing a global logical error.
The independent review in that study exposed both patterns. Of 3,851 reviewed responses, 79.79% of AI evaluations were judged fully correct, 9.55% acceptable, and 10.67% incorrect. Beyond OCR-related problems, reviewers identified over-penalization of valid methods that were not explicitly listed in the rubric and undue leniency toward rubric-shaped explanations with incorrect reasoning. Strong score alignment does not reveal which of these cases is present.
The third unanswered question concerns feedback. A comment can be fluent, detailed, and encouraging while failing to identify the reason a student’s work is unsound. In the data-analytics study, DeepSeek-R1 achieved the highest combined automated similarity scores against instructor feedback in both practices. Yet all evaluated models generated much longer feedback than instructors, and the authors explicitly warned that semantic similarity and verbosity do not automatically equal pedagogical quality. In a stratified review of 243 cases, at least one instructor judged 194 cases—79.8%—to be informative but lacking contextual depth. Reviewers identified missed minor but important mistakes in 148 cases and missed subtle methodological arguments in 189.
That is the difference between describing visible features of an assignment and diagnosing what those features mean. A model may recommend clearer variable names, more robust statistics, or better visualizations while failing to notice that a data-cleaning error invalidates the entire comparison. It may produce a useful-sounding paragraph that does not explain the consequence of the student’s mistake. A teacher score that happens to match cannot settle whether the feedback teaches the right lesson.
Finally, agreement says little about whether the system is appropriate for the stakes. The calculus deployment focused on low-stakes quizzes, and the authors identify midterms and finals as requiring substantially more conservative human-in-the-loop controls. The data-analytics study likewise treats model outputs as instructor-reviewed drafts. Before an institution expands use, it must consider review triggers, uncertainty, privacy, data handling, cost, and accountability. A system can be close enough for formative assistance while still being unsuitable as the sole judge of a consequential grade.
The diagram captures the central asymmetry. Teacher agreement is one branch leading to operational fit. A responsible deployment decision also depends on whether the work was read faithfully, whether the judgment is correct and fair, and whether the feedback is useful. Those branches can converge, diverge, or remain partly unknown.
This is why the question “How often does the AI agree with the teacher?” is a beginning rather than an ending. The useful follow-up questions are more demanding: What did the system actually see? Which rubric defined success? How did independent reviewers judge the explanation? What happens in the tails of the error distribution? Which cases are sent to a human? And what claim is the evidence strong enough to support?
The rest of this series develops those questions without discarding the practical value of teacher agreement. The goal is not to replace one simplistic authority with another. It is to understand which witness can speak to which property of an AI grading system—and what remains unresolved when the witnesses agree.