Overview

Instruction Hierarchy training uses context synthesis to distribute compatible request pieces across privilege levels while preserving the useful answer, and context ignorance to target the answer as if hostile content had no control. The paper applies these methods across open- and closed-domain tasks, browsing transfer, and extraction, with refusal when safe continuation is impossible. Synthetic generation and held-out tests limit the conclusions.

Learning outcomes

Context synthesis teaches that compatible lower-level constraints can shape an answer without overriding the higher-level purpose.
Context ignorance trains the model to preserve the task as if a hostile command had no control, refusing when safe continuation is impossible.
Synthetic prompts, model-based checks, and held-out tool or jailbreak tests provide bounded evidence rather than proof of universal robustness.

Keywords

Instruction HierarchyContext SynthesisContext IgnoranceDistillationSynthetic DataRefusal

Teaching the Boundary: Context Synthesis, Context Ignorance, and Refusal

The instruction hierarchy describes a behavior a model should learn, but a description alone does not make that behavior dependable. This chapter focuses on the paper’s training method: how synthetic examples teach a model to incorporate compatible lower-level constraints, ignore hostile content, and refuse when safe continuation is impossible. Understanding these two complementary methods also explains why the study separates open-domain assistants from closed-domain tasks and why some attack types are deliberately withheld from training.

Context synthesis preserves compatible requests

The first training problem is easy to state and easy to mishandle: how can a model learn that lower-priority instructions are sometimes valid? If every example pairs a user or tool message with a refusal, the model may learn the wrong lesson. It may become resistant to attacks by becoming resistant to users.

The paper addresses this problem with a method it calls context synthesis. It begins with a compositional request—one containing several compatible requirements. “Write a twenty-line poem in Spanish” combines a task, a language, and a length. A generator decomposes the request into smaller pieces: write a poem, use Spanish, and use twenty lines. Those pieces are then placed at different levels of the instruction hierarchy. The model is trained to produce the same ground-truth response it would have produced if the complete request had been supplied as a high-priority instruction.

The causal idea is important. The method does not teach that a user message can overrule a system message. It teaches that a user-level detail can still constrain the answer when it does not conflict with the system’s purpose. Privilege determines who wins if there is a conflict; it does not determine whether compatible information can be used at all.

Suppose a system message defines an assistant as a car-sales helper. A user asks for the best family car in a given price range and asks for the answer in Spanish. The role and the language request can coexist. The higher-level instruction establishes the domain and boundaries; the lower-level instruction fills in a preference or output constraint. Context synthesis creates training situations in which those pieces are separated but the target remains the complete, useful answer.

This is synthetic data generation: examples are constructed rather than collected as a record of naturally occurring conversations. The paper combines it with context distillation, a broader pattern in which a model is prompted with the desired context or rule and the resulting behavior is used as a training target for a model that will later operate without that extra support. The method can therefore teach a relationship that is difficult to specify with a single verbal reminder.

The relationship has two failure directions. If the lower-level pieces are placed without regard to the higher-level purpose, the training data could reward unauthorized overrides. If the data treats every lower-level instruction as hostile, it could create over-refusal. The paper explicitly describes the need to balance prompt-injection resistance against the damage caused by refusing aligned requests. Context synthesis addresses the second risk by making compatibility part of the target behavior.

Published by DeepMox

FAQ

How does context synthesis teach compatible requests?

Context synthesis decomposes a compatible request, places its pieces at different privilege levels, and trains on the original useful response.

What is context ignorance used for?

Context ignorance trains the model to answer as if a hostile lower-level instruction had not controlled the task, using refusal when safe continuation is impossible.

Why were tool-use injections and jailbreaks withheld from training?

They were withheld to test whether the model learned a broader authority relationship rather than memorized attack patterns; success supports transfer only across selected evaluations.

Previous8 minInstruction Hierarchy: Source to Consequence

Next10 minInstruction Hierarchy: Robustness Results

Instruction Hierarchy: Training the Boundary

Teaching the Boundary: Context Synthesis, Context Ignorance, and Refusal

Context synthesis preserves compatible requests

FAQ