Overview
The Instruction Hierarchy frames prompt injection as an authority problem: models should preserve higher-level intent, use compatible lower-level details, and ignore or refuse conflicting content. In selected GPT-3.5 Turbo evaluations, hierarchy training improved resistance across several attacks and some held-out tests, while causing over-refusal in some benign attack-shaped cases. Secure agents still need identity, least privilege, runtime policy, human approval, observability, and testing.
Learning outcomes
- Instruction-hierarchy training teaches models to distinguish authorized commands from untrusted content, rather than following forceful wording.
- Selective compliance follows aligned lower-level requests, ignores hostile content when the task can continue, and refuses when safe continuation is impossible.
- In the reported GPT-3.5 Turbo evaluations, training improved several attack-resistance measures and selected held-out tests, but increased over-refusal in some benign attack-shaped cases.
- Model robustness can reduce bad plans, but secure agents still require identity checks, scoped permissions, runtime policy, human approval, observability, and testing.
Keywords
Preview every chapter · Unlock full text, audio, and video from chapter 4
PlansFAQ
What does the Instruction Hierarchy teach an AI assistant?
It teaches the assistant to preserve higher-level purpose, use compatible lower-level details, and treat conflicting or untrusted content as data or a reason to refuse.
How should an AI assistant handle commands in webpages, emails, and tool results?
It should treat embedded commands as lower-priority, untrusted content while still using relevant facts, unless a separate trusted mechanism authorizes the workflow instruction.
Does the Instruction Hierarchy make an AI agent secure by itself?
No. It improves a model’s authority judgment, but the application must still enforce identity, least privilege, action policy, confirmation, memory and rendering safeguards, and auditability.
What evidence supports Instruction Hierarchy training?
In the reported GPT-3.5 Turbo setup, hierarchy training improved several direct, indirect, extraction, and selected jailbreak evaluations, but the gains were benchmark-bound and came with some over-refusal.