Teaching a Model to Critique and Revise Itself
Constitutional AI’s first operational move is not to make a model choose between two answers; it is to ask the model to improve an answer it has already produced. This chapter follows that supervised critique–revision loop, focusing on how a natural-language principle becomes a training example and what survives each transition. The practical judgment is deliberately narrow: a better revision can change the policy’s starting distribution, but it does not by itself prove that the model understands harm or that its written critique is a faithful explanation of its decision.
A response enters a supervised safety loop
The process begins with a helpful-only assistant and a prompt selected because it is likely to elicit harmful behavior. In the paper’s illustrative case, the assistant gives an answer that would facilitate an intrusion into someone else’s property. The details of that example are less important than its position in the pipeline: the initial response is a sample of the model’s existing behavior, not a rule the researchers want to teach.
A constitutional principle is then inserted into the conversation through a critique request. The model is asked to identify specific ways its last response may be harmful, unethical, dangerous, illegal, toxic, racist, sexist, or socially biased. The request changes the model’s immediate task. It is no longer merely continuing the user’s conversation; it is inspecting an assistant response against an explicit standard.
The next prompt asks for a revision that removes the identified harmful content. Ideally, the output preserves whatever can still help the user while removing assistance that would create harm. A response to a dangerous request might therefore refuse the harmful action, explain the relevant concern, and offer a lawful or safer alternative. The point is not to teach a universal refusal phrase. It is to use a principle to alter the response while keeping the interaction intelligible and, where possible, useful.
The revised answer is formatted with the original prompt as a new prompt–response pair. That detail makes repetition possible. Another principle can be sampled, the revised response can be critiqued again, and a further revision can be generated. The loop is not a sequence of independent safety filters attached at inference time. Each pass creates a new textual state that becomes the input to the next pass.
This state model highlights what persists and what changes. The user’s prompt persists as the context for the response. The assistant output changes at every revision transition. The principle is not stored as a numerical reward at this stage; it acts as the instruction that conditions the critique and rewrite. Once the prompt and revised response are selected for training, the intermediate critique may be useful as a trace of the transformation, but the supervised policy is trained on response behavior rather than granted a formal moral theorem.
The researchers used a collection of principles rather than one repeated sentence. Some addressed harmful or illegal assistance in general; others focused on social bias, age appropriateness, empathy, or avoiding offensive and accusatory language. Principles were sampled at each step. This provides two kinds of coverage. It exposes a response to more than one way of noticing a problem, and it makes the resulting revised examples less uniform. A single rule might push a model toward a consistent but narrow style; varied rules can create a broader set of safe responses.
The model must also know which role it is playing at each turn. In a long context, it can confuse a request for a critique with a request for a revision, or produce a critique when the training example requires the final answer. CAI addresses this with few-shot examples: the model sees demonstrations in which a prompt is followed by a critique request, a critique, a revision request, and a revision, all in the same format. These examples are not ornamental. They establish the boundaries between inspecting an answer and producing the answer that will be shown to the user.
That boundary is important for a generalist reader because “self-critique” can sound more capable than the mechanism warrants. The model is not stepping outside its learned behavior to consult an independent conscience. It is being prompted to generate one kind of text after another. Its critique can still be sensible and useful; the paper reports that revisions often remove much of the harmful content. But the reliability of the result comes from the interaction of prompting, examples, model capability, and later training—not from the existence of a private evaluator separate from the model being revised.
There is a second reason to distinguish the steps: they have different control surfaces. The principle controls what the model is asked to notice. The critique controls which problem is made salient in the next prompt. The revision controls the candidate behavior that can enter supervised learning. The fine-tuning process then changes the policy’s future probabilities. A weakness at one surface need not look like a weakness at the next. A critique can name the wrong issue while still nudging the revision away from dangerous assistance; conversely, a well-written critique can be followed by a poor rewrite. Treating the loop as a single act of “reflection” hides these failure paths.
The choice to begin with a helpful-only assistant is part of the same design. The researchers are not asking a blank model to invent a safe conversational style from a list of principles. They are using an assistant that already has instruction-following behavior and then redirecting selected responses. That inherited capability supplies the language, context handling, and willingness to engage that a safety-only intervention might otherwise suppress. CAI’s supervised stage is therefore a controlled transformation of an existing policy, not a complete alignment procedure built from rules alone.
Repetition changes the training distribution
After generating revisions, Anthropic fine-tuned a pretrained language model on them. The supervised data also included responses from the helpful RLHF assistant on ordinary helpfulness prompts. That second stream protects an important property: the model should not learn harmlessness only by becoming silent or unresponsive. The desired starting policy must learn both to avoid harmful assistance and to continue attempting legitimate tasks.
The scale of the data illustrates why the method is more than a handful of hand-written examples. The authors used 42,496 human-written red-team prompts and generated a further 140,335 prompts with a pretrained model, for a total of 182,831 red-team prompts. They sampled four critique–revision pairs per red-team prompt. The helpfulness collection contained 135,296 human-written prompts, with two responses sampled per prompt. These figures describe the research setup, not a universal recipe, but they show the role of automated generation: once the principle and format are specified, a model can produce a large body of candidate training material.
The revisions are not treated as equally informative at every depth. When the researchers scored initial responses and successive revisions with a preference model trained on human feedback, harmlessness and combined helpfulness–harmlessness scores rose with the number of revisions. Pure helpfulness scores fell. The first pass often removed the most obvious harmful content; later passes tended to make smaller changes. This is a familiar systems trade-off: repeated optimization can push one metric in the desired direction while eroding another property that was not the immediate target.
The score trend also needs a qualification. The paper warns that preference-model scores become less calibrated at higher values. A monotonically rising score does not necessarily mean that each textual revision represents an equally large real improvement. Once a response is already judged relatively harmless, the evaluator may have less resolution for distinguishing a thoughtful refinement from a merely more cautious or formulaic answer. Repetition changes the distribution of training examples, but the metric used to observe that change can itself lose precision.
The number of principles has a different effect. In the reported comparison, adding more principles did not appear to raise the harmlessness preference-model score directly. The authors nevertheless expected a larger collection to produce more diverse revisions, and diversity matters for later reinforcement learning because the policy needs varied behavior to explore. This is a useful distinction between a direct outcome and an enabling condition. A wider set of principles may not make each individual response score higher, yet it can change which alternatives the next stage gets to see.
The helpfulness data supplies another form of diversity. A safety-only corpus would make the policy’s world smaller: it might learn that the safest response is usually to decline, regardless of whether the user’s request is benign, ambiguous, or dangerous. By mixing helpfulness samples into supervised fine-tuning, the researchers try to preserve instruction-following behavior while shifting the response distribution away from harmful assistance. The result is intended as a starting policy for reinforcement learning, not the final model.
That starting-policy role changes how the data should be judged. A revised answer does not need to settle every possible edge case in order to be useful to the next stage. It needs to move the model away from obvious harmful behavior while leaving enough legitimate variation for later comparison and optimization. At the same time, a training set that contains only easy cases can create false confidence. The model may learn the visible style of a safe answer without learning how to distinguish a genuinely dangerous request from a benign question about the same topic. The paper’s red-team construction is intended to expose harmful behavior, but its authors do not claim that the resulting distribution exhausts real-world attacks or contexts.
The paper’s comparison between critiqued and direct revisions shows why the intermediate step is a design choice rather than an unquestionable necessity. In the direct variant, the model is simply asked to rewrite the original response according to a principle, without first generating a critique. For smaller models, critiqued revisions generally achieved higher harmlessness scores. For larger models, the two approaches performed similarly, although the critiqued version remained slightly better in the reported tests.
Why keep the critique if a direct rewrite can work? One reason is diagnostic visibility. A critique gives a reader or developer a textual account of what the model noticed before it changed the answer. It may also prompt the model to consider subtle harms or unintended consequences that a direct rewrite would overlook. But the same paper reports that the critiques were sometimes inaccurate or overstated. That means the critique has two different statuses: it can be a useful causal intervention in the generation process, and it can be an unreliable explanation of why the final answer is preferable.
The distinction resembles a familiar engineering situation. A system can produce a better output after an intermediate check even when the check’s log is incomplete or occasionally wrong. The check changes the path the system takes; it does not automatically provide a trustworthy account of the system’s internal state. CAI’s supervised loop should therefore be credited with behavioral bootstrapping and possible transparency, not with proven interpretability.
There are also clear conditions under which the loop can fail. If the principle is vague, the critique may focus on style while missing a consequential harm. If the model cannot maintain the critique/revision roles, formatting errors can enter the training set. If revisions are sampled from a narrow prompt distribution, the resulting policy may not generalize to harms outside that distribution. If helpfulness examples are too weak or safety pressure is too strong, the policy may become less capable or more evasive. The paper’s evidence supports these as design boundaries, not as problems solved by adding another pass.
The supervised stage nevertheless performs a strategically important transformation. It takes a helpful policy that may produce harmful answers and moves many of its samples toward responses that are safer, more thoughtful, and less likely to shut down the conversation. That gives the later RL stage a more favorable starting point and a broader set of candidates to explore. What it does not yet provide is a learned function for choosing between two revised responses across new prompts. That is the next unresolved boundary: whether principles can guide comparisons well enough to become a preference signal.