When the Same Score Means Different Things

The most revealing AI-grading failures are not always the largest score gaps. A large gap can be explained by two defensible rubrics, while an exact match can conceal different reasoning or shallow feedback. This chapter examines documented calculus and data-analytics cases to show how scoring resolution and contextual judgment shape apparent agreement. The practical skill is to inspect what a score represents before calling it a success or a failure.

In calculus, disagreement can be a resolution problem

The UC Irvine calculus system used two complementary rubrics because handwritten solutions can be correct in more than one way. The flexible rubric evaluated global mathematical coherence and allowed diverse approaches. The fixed rubric decomposed a canonical route into required components and common-error checks. Both designs have a purpose. Flexibility protects a student whose reasoning is valid but does not resemble the example. Fixed criteria make expectations visible and reduce the chance that a required step disappears inside a general impression.

The two rubrics do not measure exactly the same thing. Consider a series problem in which a student identifies the correct geometric structure, uses an equivalent form for the first term, and reaches the correct sum, but writes the indexing in a nonstandard way. The flexible rubric can recognize that the expressions are algebraically equivalent and that the reasoning is coherent. The fixed rubric can treat the unclear indexing as a partially satisfied criterion. In the study’s representative example, the flexible rubric awarded full credit while the fixed rubric awarded partial credit.

The difference is not merely a matter of the language model being lenient. It is a difference in what counts as adequate evidence of understanding. The flexible rubric asks whether the solution works as a whole. The fixed rubric asks whether the student has displayed a set of expected components clearly enough to receive each point. A teacher who gives full credit may be emphasizing mathematical equivalence; a rubric that gives partial credit may be emphasizing explicit communication of the canonical structure. Both priorities can be educationally defensible, but they should not be mistaken for one shared measurement.

The reverse can happen too. The calculus study notes that a flexible rubric may give a lower score than a fixed rubric when locally correct steps become globally inconsistent or numerically wrong. A checklist can see that a student wrote down a ratio, a first term, and a summation formula, then award component points even when those pieces do not fit together. A globally oriented evaluator may detect that the conclusion does not follow. Flexibility is therefore not synonymous with higher grades, and fixed structure is not synonymous with accuracy.

The researchers evaluated both rubrics through separate model calls and used a max-rule: the higher score became the final score, along with its feedback. On the tested quizzes, this produced the lowest mean absolute error against TA scores. That result supports the max-rule as a useful local response to rubric variation. It does not establish that taking the maximum is always the fairest aggregation.