The Teacher Is a Reference Point, Not an Oracle
Human grades are the most relevant comparison for an AI grading system because they determine what happens in a real course. They are not automatically a perfect record of correctness. This chapter examines how to use teacher and TA judgments as a practical reference while accounting for disagreement, rubric design, and the different resolutions at which people evaluate partially correct work. The result is a more useful standard: characterize the human reference instead of treating it as an oracle.
Why open-ended work has no single score
A score on an open-ended response is not like the reading on a thermometer. The student may reach the right conclusion through an unusual but valid method, make a correct local move inside an incomplete solution, or write an answer that is mathematically sound but poorly organized. A rubric can specify how to treat these cases, but it cannot make every borderline judgment inevitable.
That is especially true for partial credit. A calculus solution may contain several separable decisions: choosing a method, setting up an expression, carrying out an algebraic transformation, justifying a conclusion, and reporting an answer. Two graders can agree that one step is correct while disagreeing about how much that step should contribute to the total. They may also weigh an unconventional route differently from a canonical route. The disagreement is not necessarily a failure to understand mathematics. It may reflect a different judgment about the learning objective, the importance of an error, or the granularity of the rubric.
The UC Irvine study makes this problem explicit. It notes that there is no publicly available expert-graded calculus dataset that can serve as an absolute benchmark, and that human graders show nontrivial disagreement, particularly on borderline or unconventional answers. For that reason, the study treats TA grades as a practical reference for outcome alignment rather than as unquestionable ground truth. The data-analytics study uses instructor grades and feedback as comparison benchmarks too, but adds statistical, semantic, and qualitative checks for the same reason: the reference is useful without being complete.
This does not mean that assessment has no standards. A human judgment can still be well supported by the problem statement, the stated learning objective, the rubric, the student’s reasoning, and a consistent explanation of the deduction. “No unique score” means that several judgments may be defensible within a range; it does not mean that every score is equally good. The validation task is to discover where the range lies, why judgments differ, and whether the difference matters.
The study’s inter-reviewer comparison provides a concrete baseline. Two independent reviewers graded the same 171 calculus question instances with the same criterion-based rubric. Exact agreement occurred for about 45% of the questions, while differences across several score levels were common. That result should be read carefully. It does not show that the reviewers were interchangeable, nor that every AI disagreement falls within an acceptable human range. It shows that a nonzero amount of disagreement is present even before an AI system enters the comparison.
This baseline changes how an AI–human gap should be interpreted. A small difference may be compatible with ordinary human variation, particularly for a borderline solution. A large difference may signal an OCR failure, a rubric mismatch, or a genuinely incorrect judgment. An exact match may be reassuring, but it can also occur because the rubric is coarse or because both evaluators overlook the same issue. The number alone cannot tell us which explanation applies.
A good evaluation therefore reports more than the average human–AI difference. It should ask how disagreement is distributed, whether it concentrates in particular problem types, and whether the disagreements affect core learning objectives or only presentation choices. A system that differs from humans on harmless notation but agrees on mathematical reasoning may be more acceptable than one that matches the average while mishandling a critical misconception.
The relevant comparison is also conditional on the decision being made. A reviewer disagreement about whether an equivalent algebraic form deserves full or nearly full credit may be tolerable in a formative quiz if the feedback makes the equivalence clear. A disagreement about whether a student used a valid theorem, understood a key concept, or produced an invalid analytical conclusion is more consequential. The same numerical distance can therefore carry different educational and institutional risk. A useful human baseline should classify disagreements by their reason and consequence, not merely place them inside an average range.
Human variation cannot become a blanket excuse for model error. If two reviewers disagree because the rubric is underspecified, the result points to a calibration or rubric problem. If they disagree because one overlooked a critical logical flaw, the range should not be used to excuse that oversight. The purpose of measuring human variation is to make these possibilities visible and decide which cases require adjudication, not to lower the standard until every output looks acceptable.
Rubrics create the object being compared
A rubric is not just a measuring tape placed beside an already existing score. It defines which features of the work count, how much they count, and what kinds of reasoning deserve credit. When an AI score is compared with a teacher score, the comparison is partly between two applications of a construct shaped by the rubric.
The calculus system used two complementary designs. Its flexible rubric was intended to evaluate global mathematical coherence and credit diverse valid approaches, including noncanonical reasoning. Its fixed rubric decomposed the expected solution into point-based checks for required steps and common errors. Each could see something the other risked missing. The flexible rubric could recognize an equivalent expression or a valid reindexing that a checklist did not anticipate. The fixed rubric could make missing components explicit and produce a more consistent point map.
The researchers evaluated both rubrics and selected the higher score through a max-rule. On the tested Math 2A and Math 2B quizzes, that rule produced the lowest mean absolute error against TA scores. The result supports the rule as a useful local design choice, not as a universal law. The researchers did not exhaustively compare it with every alternative aggregation method, and a rule that fits one course’s grading culture may not fit another’s.
The reason becomes clear in a representative calculus case. A student used a noncanonical but mathematically correct route to sum a series. The flexible rubric recognized the coherent reasoning and algebraic equivalence and awarded full credit. The fixed rubric assigned partial credit because the student’s indexing and notation did not fit its canonical point map as clearly. Neither rubric was simply “seeing the facts” independently of interpretation. They emphasized different evidence of understanding.
The opposite tension is equally important. Flexibility does not guarantee generosity in every case. The calculus study notes that a flexible rubric can assign a lower score than a fixed rubric when locally correct steps become globally inconsistent or numerically wrong. A checklist may award points for visible components even when the argument they form is unsound. This is why “the AI uses a more flexible rubric” cannot be translated into either “the AI is fairer” or “the AI inflates grades” without inspecting the actual reasoning.
A second case shows how grading resolution affects an apparent conflict. In one example, the AI assigned a much higher score than the TA because the AI rubric awarded partial credit across several criteria while the TA used coarser score bins. The study judged both scores defensible under their respective rubrics. The disagreement was not evidence that one evaluator had necessarily misread the mathematics. It showed that the two systems had different ways of representing incomplete understanding.
This distinction matters for validation design. If a study treats the TA score as the only correct label, it may penalize an AI for expressing partial understanding at a different resolution. If it treats the AI’s finer rubric as automatically superior, it may reward a model for making more deductions look precise without proving that those distinctions matter educationally. The proper question is whether the rubric captures the intended learning construct and applies it consistently to valid variation.
The data-analytics cases extend the same logic beyond mathematics. Open-ended Python notebooks require judgments about code behavior, data cleaning, analytical method, visualization, documentation, and interpretation. A model can assign a score close to the instructor’s while giving generic advice that misses a methodological flaw. Conversely, a model may disagree numerically because it weights one dimension differently, even while identifying a real issue. Here too, the rubric and the explanation must be inspected together.
What a human reference can legitimately do
Teacher and TA scores have at least three legitimate roles in validation. First, they are an operational anchor. If an assistant is intended to support an existing course, its outputs should be compared with the people who currently assign the grades. Large systematic differences are a reason to investigate before deployment, not something a model’s abstract benchmark score can erase.
Second, human grades are a calibration anchor. They reveal local expectations about partial credit, acceptable notation, methodological rigor, and the relative importance of errors. A model that matches a reference only after a prompt or rubric change may have improved its fit to the course. That improvement is valuable, but it should be described as calibration to a defined practice rather than proof of universal correctness.
Third, instructors provide pedagogical context that an aggregate label cannot contain. They can explain why a minor-looking error invalidates a later conclusion, why a false start should not be penalized after a student corrects it, or why a nonstandard method demonstrates the intended concept. Those explanations are not incidental commentary; they are part of the evidence needed to judge whether a score represents the learning objective.
Independent reviewers add an important but limited fourth perspective. They reduce the circularity of asking whether an AI agrees with the same teacher whose rubric shaped the system. In the calculus study, reviewers examined input quality, score alignment under the study rubric, rubric fidelity, and feedback accuracy. Their review exposed over-penalization of valid methods and undue leniency toward familiar but incorrect reasoning. That is more informative than a score-gap histogram alone.
Still, independent reviewers remain human and remain connected to the rubric they are asked to use. Their judgments can differ from a TA’s without proving that the TA was wrong, and they can agree with the AI while sharing a blind spot. Independence makes the comparison more informative; it does not transform the reviewer into an infallible ground-truth generator.
The sequence matters. The evaluator should not ask only whether the AI and teacher returned the same number. They should ask what each evaluator observed, which rubric each applied, how much human variation exists, and whether the disagreement changes the educational meaning of the result. An apparently successful match is stronger when the input is legible, the rationale is coherent, and independent review finds no material issue. It is weaker when only the final number is available.
This gives the human reference a disciplined role. It is not an oracle whose every label ends the inquiry, and it is not noise to be discarded in favor of model output. It is a situated account of how a course currently understands and rewards student work. Validation becomes more credible when that account is made explicit, its variation is measured, and its blind spots are tested.
The next step is to connect this reference problem to the full path from student work to automated score. A human comparison occurs at the end of that path, but the decisive error may have entered much earlier—in what the system read, what the rubric represented, or what the model inferred from the representation.