What the Results Actually Show: Robustness, Generalization, and Over-Refusal

The instruction hierarchy makes an empirical claim: training a model to respect message authority can improve resistance to prompt attacks without destroying ordinary instruction following. This chapter examines what the reported evaluations actually test—the size of the robustness gains, the evidence from attacks withheld during training, and the cost of over-refusal. The goal is a calibrated judgment about the experiment’s evidence, not a promise that a benchmark score turns a probabilistic model into a security boundary.

The experiment tests more than whether the model says no

The study fine-tuned GPT-3.5 Turbo with supervised fine-tuning and reinforcement learning from human feedback. The training mix included the instruction-hierarchy examples described earlier as well as data for general model capabilities. The baseline was similarly fine-tuned on capability data but did not receive the hierarchy examples. For both models, the researchers selected the best-performing checkpoint according to validation accuracy and then evaluated safety and capability behavior.

That comparison matters because it asks whether the hierarchy data changes behavior beyond ordinary post-training. It does not compare every possible model or training method. It is a controlled test of one model family, one data-generation pipeline, and one set of evaluation tasks. The result should therefore be read as evidence about that setup, not as a universal property of instruction-hierarchy training.

The evaluation suite has three different jobs. Attack evaluations ask whether the model resists prompt hijacking, conflicting user instructions, indirect injections, or extraction. Capability evaluations ask whether general performance remains comparable. Over-refusal evaluations ask whether the model still follows benign or borderline requests that resemble attacks. A single robustness number cannot answer all three questions because a model could improve its attack score simply by refusing everything.

The main attack results show broad improvement, not just a fix for one phrase. On the TensorTrust prompt-hijacking evaluation, robustness rose from 59.2% for the baseline to 79.2% for the hierarchy-trained model. On new direct prompt injections, the increase was smaller but still positive, from 89.6% to 93.7%. For user-conflicting instructions, the change was from 62.2% to 92.6%. Indirect prompt injections placed in browsing results rose from 77.5% to 85.0%.

The most dramatic displayed change concerns system-message extraction: the baseline scored 32.8%, while the hierarchy-trained model scored 95.9%. The paper summarizes the main results as improvements of up to 63 percentage points. Taken together, the values suggest that the training examples affected several related attack surfaces, including direct conflict, indirect content, and protected instructions. The qualitative cases support the same interpretation: the trained model translated injected text as data, refused an attempt to reveal a password, and declined an instruction that tried to take over a constrained assistant.

The numbers are not interchangeable. Prompt hijacking measures whether a model produces an attacker’s target phrase.