A Score Is the End of a Chain

The evidence so far can be joined into a single way of looking at AI grading: a score is the end of a chain, not a direct reading of student understanding. This chapter connects the difference between alignment and correctness with the limits of human references. The practical judgment is simple to state but demanding to apply: before accepting an AI–TA agreement result, trace what the system saw, what instructions shaped its interpretation, and which part of the final output the evidence actually validates.

From student work to a grading decision

The UC Irvine system makes the chain visible because its input is handwritten. A student’s page is not sent straight from paper to score. The system first segments the relevant answer regions, then transcribes handwriting into LaTeX, then supplies that representation—along with problem context, reference solutions, and structured rubrics—to a language model. The model produces a numerical grade and explanatory feedback. Each stage changes the form of the evidence available to the next one.

The segmentation stage determines which marks belong to which problem. The transcription stage determines which symbols, words, cancellations, and relationships are made available for reasoning. Contextual prompts can help a model interpret ambiguous handwriting, but they also create a boundary between what the student visibly wrote and what the model infers from the problem statement. The rubric and reference solution then determine which features of the interpreted response count toward the grade. Finally, the score and feedback compress a long chain of observations and judgments into outputs that can be compared with a TA’s result.

That sequence matters because the final comparison is downstream of all the earlier transformations. If two evaluators produce the same score, they may have agreed about the student’s work, or they may have arrived at the same number after different readings and different weighting choices. A final score can be aligned while the path that produced it remains unknown.

flowchart LR A["Student work"] --> B["Region segmentation"] B --> C["OCR and transcription"] C --> D["Problem context and reference solution"] D --> E["Structured rubric and prompt"] E --> F["LLM evaluation"] F --> G["Score and feedback"] G --> H["Teacher or reviewer comparison"] H --> I["Accept, inspect, or defer"]

This is not a claim that every stage fails independently or that the model cannot use context appropriately. It is a reminder that different evidence is needed to validate different transformations. A score-gap statistic evaluates the last comparison. It does not automatically validate the segmentation, the transcription, the rubric, or the explanation.

The transformations also trade information for interpretation. Literal transcription preserves what appears on the page, but it may leave an ambiguous symbol unresolved. Context-aware transcription can use the problem statement to infer whether a faint mark is likely part of a fraction or whether a misspelled word means “divergent.” That can make a downstream judgment more robust to messy handwriting.