The Assessment Signal: What Large-Scale Data Can and Cannot Tell Us

Grades are one of education’s most visible signals, but they are not a transparent window into learning. This chapter examines how the spread of generative AI interacts with assessment design, what a large university panel found about grades and course evaluations after ChatGPT, and how process-oriented evidence can make human understanding more visible. The practical judgment is deliberately bounded: a stable grade signal is worth knowing about, but it does not by itself prove that every student learned, nor does a worrying mechanism by itself prove that grades have already collapsed.

The distinction is important because an assessment has several jobs at once. It can give a student practice, show an instructor what needs teaching, certify that a learner has met a standard, and signal capability to a future instructor or employer. A polished product may serve one of those jobs while failing another. Generative AI makes that separation harder to see because it can produce credible work without revealing how much of the underlying reasoning belonged to the student.

A grade is shaped by the assessment that produces it

An assessment is more susceptible to direct AI substitution when a student can complete it independently of instructor observation and submit the result later. Take-home exams, homework, papers, and many projects fit that description. An in-person closed-book exam, a presentation, or a live demonstration is less susceptible because the student has to show performance in real time. AI may still help the student prepare for the latter, but it cannot as easily perform the assessed demonstration at the moment it is observed.

This is a statement about opportunity, not behaviour. A take-home essay can be designed to require original reasoning, drafts, discussion, and an oral defence, and a student may use no AI at all. A closed-book exam can test memorised procedures without measuring deep understanding. “Susceptible” is therefore not a moral label attached to an assignment or a discipline. It is a property of how much the assessment allows an external system to stand in for the student’s visible work.

That property is the basis of a large study by Dumlao and colleagues. The researchers assembled a balanced panel from a large Midwestern public university, covering fall 2016 through fall 2025 and comprising 122,663 students and 38,754 course offerings in the main analysis. They estimated whether the November 2022 release of ChatGPT changed outcomes differently in courses whose final grades placed more weight on assessments susceptible to direct GenAI automation.

The study’s comparison is not “students who used AI” versus “students who did not.” It compares courses with different opportunities for substitution before and after a common technology shock. That makes it useful for a system-level question: did the arrival of widely available ChatGPT show up as a differential movement in grades, withdrawals, failures, or student evaluations where direct automation was easier?

The answer is cautious. After separating the COVID-19 disruption from the AI shock, the estimated difference in average grades between fully susceptible and fully non-susceptible courses fell in a range of 0.030 to 0.045 grade points. The estimate was positive but economically small and not statistically significant at the 5% level. More susceptible courses also showed no detected differential change in withdrawal or failure rates. The data therefore do not show a universal post-ChatGPT collapse of the grade signal at this university during the first years after the release.

A descriptive increase appeared at the top of the grade distribution under one specification: the likelihood of receiving at least an A rose by 0.107.