Where Thresholds Miss the Harms They Mean to Catch

The stress tests behind the AI incident-escalation research are useful because they do not ask only whether a framework contains a serious-sounding category. They ask what happens when the category meets a real pattern: a danger appears before harm, many small events accumulate, a legal term cannot be measured, or a condition has no clear beginning or end. This chapter applies those failure modes to the threshold architecture surrounding H.R. 9477. The goal is to distinguish four different causes of under-detection rather than treating every missed case as one generic weakness.

Waiting for confirmed harm can close the intervention window

The first failure is temporal. A framework may require a realized harm before reporting or escalation, even when the developer can already observe the event that creates the danger. The research paper uses CBRN assistance and model-weight exfiltration as examples. The developer may see the capability or the theft attempt while having no visibility into the downstream user, target, or consequence. By the time harm is confirmed, the event may have moved beyond the point where intervention is useful.

The paper identifies three conditions that make a pre-harm trigger especially important. The intervention window must be narrow. The probability of severe and irreversible harm without action must be high. And no single actor can see both the risk pathway and the eventual harm. These conditions are more discriminating than a rule that simply says “report all possibilities.” They explain why a pre-harm signal deserves attention when waiting is likely to produce certainty only after the meaningful choices have disappeared.

H.R. 9477 partially addresses this gap. Its proposed reportable-activity categories include control-undermining behavior, model-weight compromise, cyber and CBRN-related capabilities, and a circumstance in which serious harm was reasonably likely but prevented only by an unrelated action, a user’s lack of capability or intent, or other fortuity. The Act therefore does not make a completed injury the only possible gateway.

That design still requires judgment. A claim that a capability could materially enable a dangerous activity must be tied to evidence and to the statutory risk conditions. A near miss that occurred because a user lacked intent is not automatically proof that a comparable event would have caused serious harm under different circumstances. The bill’s language creates room for the early signal; future rules must make the signal distinguishable from speculation.

The failure diagnosis is thus not “the system needs more aggressive escalation.” It is more precise: a confirmed-harm condition is in the wrong place for risks whose consequences occur outside the developer’s visibility or become irreversible quickly. A pre-harm gate is a repair only when the intervention-window, severity, and visibility conditions support it.

Individual events can hide a population-level pattern

The second failure is the unit of analysis. If a framework asks whether each incident is severe on its own, it can miss a pattern made of many individually sub-threshold events. The paper uses cumulative non-consensual deepfakes and psychological harm from human–AI interaction as stress tests. It reports approximately 6,700 non-consensual images per hour at peak in one case and approximately 500,000 weekly ChatGPT users exhibiting risk factors for psychosis, mania, or suicidal ideation in a developer disclosure.

Those figures are reported by the paper and rely on its cited public materials; they were not independently reverified in this run. Their value here is not to settle the underlying estimates. It is to expose a measurement problem. One image may be classified as a low-severity event even while thousands of images produced through the same feature create a large-scale dignity and safety problem.