When Principles Become Training Data
Constitutional AI (CAI) is Anthropic’s attempt to train a language assistant to be both useful and less harmful by steering it with a written set of principles. The important question is not whether a short list of rules can somehow make a model moral. It is how those principles are converted into critiques, revised answers, comparisons, rewards, and eventually changes in model behavior. This series follows that conversion so readers can judge what CAI actually improves, what its evidence does not establish, and which responsibilities remain human.
That framing matters because a language model does not begin with the objective “help this person in a truthful, safe, and context-sensitive way.” Its pretraining objective is to predict the next token from a large body of text. The resulting system may know how an instruction-following assistant sounds, but knowing a pattern is not the same as reliably pursuing the user’s intent. A model can produce an articulate answer that follows the surface of a request while inventing facts, amplifying a stereotype, or giving advice that creates real-world harm.
CAI is a response to that gap. It is also a response to a practical problem in alignment research: human judgment is expensive, difficult to summarize, and hard to scale across every situation a general-purpose assistant may encounter. The method asks whether a model can help generate some of the supervision needed to train another model, provided that the desired direction is made explicit in a constitution.
The subject is a pipeline, not a rulebook
In the CAI paper, the constitution is a short collection of natural-language principles accompanied by examples. The principles ask a model to remove harmful, unethical, illegal, toxic, racist, sexist, or socially biased content, but they also ask it to remain helpful, honest, polite, thoughtful, and non-evasive. That combination is significant. A system that responds to every difficult question with a refusal may avoid some obvious harms, but it is not a useful assistant and may conceal rather than explain its judgment.
The principles therefore do not function like a complete legal code that determines one answer for every prompt. They act more like a steering interface. They tell a model what to look for and what direction to prefer, while leaving the model to interpret words such as “harmful,” “helpful,” or “overly reactive” in context. Anthropic reports using 16 principles in this research and sampling among them during training. The authors also describe those principles as ad hoc and say that future versions should be developed with broader stakeholder input and adapted to the setting in which a model will be used.
That qualification is central to understanding the subject. A constitution can make an objective more visible than a huge, private collection of preference labels, but visibility is not completeness. The fact that a rule is written down does not settle conflicts between rules, specify every affected group’s interests, or predict how the rule will behave after optimization. CAI’s innovation lies in what it does with the principles after they are written.
A useful high-level map looks like this:
The upper path is the supervised stage. A helpful-only assistant first responds to prompts designed to elicit harmful behavior. The model is then asked to critique its answer according to a principle and revise it to remove the problem. Those revised responses become supervised training examples. The lower path is the reinforcement-learning stage. The model generates pairs of responses, and a feedback model chooses between them under a constitutional principle. Those choices train a preference model, which supplies the reward used to improve the policy.
This is why a principle has to become data and a reward before it can reliably shape a model. A sentence in a document does not alter the probability distribution of future responses by itself. It must enter a process that repeatedly exposes the model to examples, comparisons, or consequences. In CAI, the same general idea is operationalized twice: first as a way to create better examples, then as a way to create the preference signal used by reinforcement learning.
Why the distinction from RLHF matters
CAI is easier to understand when set beside reinforcement learning from human feedback, or RLHF. In the InstructGPT pipeline described by OpenAI, humans first write demonstrations of desirable behavior. A supervised model learns from those demonstrations. Humans then rank several model outputs, a reward model learns to predict those rankings, and reinforcement learning adjusts the policy toward higher predicted reward. A KL penalty keeps the policy from drifting too far from its supervised starting point; a mixture of pretraining updates can help limit losses on capabilities that the reward data does not represent well.
CAI keeps this architecture but changes the source of one important channel of feedback. In the reported experiments, human judgments still supply the helpfulness data. Direct human labels for harmlessness are replaced by model-generated comparisons guided by constitutional principles. The preference model is consequently hybrid: it learns from human-helpfulness comparisons and AI-generated-harmlessness comparisons. Reinforcement learning still consumes a learned reward, not the constitution as a raw text file.
That difference makes the two stages complementary rather than interchangeable. The supervised stage moves the initial policy onto a more useful distribution. It gives reinforcement learning safer material to explore and reduces the chance that the later optimization process begins by producing only the most obviously harmful responses. The RL stage then compares alternatives and pushes behavior toward the response the feedback system prefers. In the paper’s description, the supervised phase gets the model “on-distribution,” while the RL phase refines performance and reliability.
The arrangement also explains why “human-free harmlessness” is a misleading shorthand if it is taken too literally. CAI removes direct human harmlessness labels from the demonstrated training loop. It does not remove the people who decide which principles to write, which examples to provide, which prompts to use, how to combine harmlessness with helpfulness, or how to evaluate the final behavior. Human feedback is reduced in one channel and relocated in others; it is not replaced by a value-neutral machine.
The companion RLHF study makes the reference group visible. Its models were trained from demonstrations and comparisons produced by a relatively small contractor pool, under instructions written by researchers, on prompts shaped by customers’ use of an API. The paper reports substantial agreement among labelers, but it also emphasizes that labelers, researchers, customers, end users, and people affected by deployments may have different interests. A system can therefore be well aligned with the feedback process that produced it without being aligned with everyone who encounters its outputs.
What CAI promises—and what it does not
The strongest promise in CAI is operational. If capable models can produce useful critiques and comparisons, developers may be able to spend less human effort collecting harmlessness labels and more effort designing clear principles, testing difficult cases, and supervising the overall process. The paper reports that model-generated critiques and revisions reduce harmfulness in the supervised data, and that RL-CAI improves harmlessness relative to the comparison models while avoiding the highly evasive behavior seen in earlier harmlessness-trained assistants. The authors also find that chain-of-thought-style reasoning can improve model performance on comparison tasks, at least in their evaluations.
Those findings matter, but they are not a safety certificate. The evaluations are built from particular red-team prompts, comparison instructions, model snapshots, and human judgments. “Harmlessness” is an operational category covering selected forms of physical, psychological, social, and other harm; it is not a guarantee against every misuse or failure. The authors report that critiques can be inaccurate or overstated, that additional revisions can lose calibration, and that reinforcement learning can produce overreaction or repetitive boilerplate when the reward is optimized too aggressively. A model may therefore sound more constitutionally compliant while becoming less context-sensitive.
The same principle-to-feedback pipeline that makes supervision cheaper can amplify an omission. If a principle overlooks a kind of harm, the generated examples may underrepresent it. If a feedback model confuses confidence with correctness, the preference model can turn that confusion into a reward. If evaluators reward refusal more readily than explanation, the policy may become evasive; if they reward confident moralizing, it may become preachy. Each stage can improve the measured target while changing the behavior people actually care about.
The practical judgment supported by this series is therefore conditional. CAI demonstrates that written principles can generate useful training feedback and can partially replace direct human harmlessness labeling in a reported research setup. It does not show that principles are complete, that model reasoning is transparent merely because it is written out, or that optimization has escaped human values. To assess CAI responsibly, readers must keep three questions separate: Did the policy behave better on the tested distribution? Was the feedback signal well calibrated and context-sensitive? And who had the authority to define what “better” meant?
Those questions lead naturally to the next step. Before examining how CAI generates critiques and revisions, we need to understand the feedback architecture it inherits: why a pretrained model requires demonstrations, a reward model, and constrained optimization in the first place. Only then can the constitutional change be seen clearly—not as a magical rulebook, but as a new way of producing the data and rewards that shape an assistant.