Overview

The supervised stage of Constitutional AI does not copy a constitution into model parameters. It turns principle-guided critiques and revisions, mixed with helpfulness examples, into a policy that is more likely to produce safer behavior and a broader set of candidates for later reinforcement learning. The reported evidence supports movement in measured harmlessness and some inspectability, but pure helpfulness can decline, critiques can be inaccurate, and the loop does not establish robust generalization, faithful reasoning, or a reliable comparison function.

Learning outcomes

A principle, critique, revision, and supervised policy are different objects with different failure points.
Supervised CAI changes the starting distribution and can diversify later exploration; it does not supply a reliable chooser between answers.
Critiques make stated reasons inspectable, but inaccurate or overstated text is not proof of faithful reasoning.

Keywords

Constitutional AIsupervised learningstarting policyexplorationcritiquehelpfulness

The First Loop: What Supervised CAI Actually Changes

The supervised stage of Constitutional AI connects a written principle to a revised answer and then to a new starting policy. The useful judgment here is not whether that loop “solves” harmlessness, but which properties it changes before reinforcement learning begins: the policy’s response distribution, the variety of behavior available for exploration, and the visibility of the model’s stated reasons. Keeping those properties separate makes it possible to see both the value of the loop and the exact question it leaves open.

A single pipeline contains four different objects

The easiest way to misunderstand supervised CAI is to treat the constitution, the critique, the revision, and the trained policy as interchangeable forms of the same thing. They are not. They occupy different stages and carry different kinds of authority.

The constitutional principle is a human-written instruction about what should count as a problem or what kind of answer should be preferred. It can direct attention toward dangerous assistance, social bias, age appropriateness, empathy, or an unnecessarily accusatory tone. It is a steering input. It does not itself show how a model will interpret the instruction when a prompt is ambiguous or when several values conflict.

The critique is a model-generated assessment of an existing response under that instruction. It identifies what the model believes is harmful or inappropriate. Because it is written in natural language, it is easier for a person to inspect than a change in millions of model parameters. But it remains an output sampled from the same kind of system that produced the original answer. It can be reasonable, incomplete, inaccurate, or overstated.

The revision is a new candidate response conditioned by the critique request. It is the object that most directly changes what a user would see. A good revision removes harmful assistance without turning every sensitive question into a blank refusal. It may explain the concern and offer a safe alternative, preserving some of the helpfulness that motivated the assistant in the first place.

The supervised policy is different again. It is trained on prompt–revision examples, together with helpfulness examples, so that future responses are more likely to resemble the revised behavior. The policy does not receive a permanent copy of the constitution as a guaranteed rule interpreter. It learns a distribution over responses shaped by the examples it was shown. In the training setup, the intermediate critique helps produce the revision, while the revised response becomes the principal behavioral target.

The supervised loop can be summarized as a boundary map:

flowchart LR P["Constitutional principle"] --> E["Critique and revision examples"] E --> S["SL-CAI starting policy"] S --> B["Safer starting distribution"] S --> D["More varied exploration"] B --> Q["Unresolved: can AI compare responses reliably?"] D --> Q S --> R["Unresolved: robust beyond tested prompts?"]

This map clarifies what crosses each boundary. A principle crosses from human design into a model prompt.

Published by DeepMox

FAQ

What changes in the policy before reinforcement learning?

The starting policy shifts toward safer responses and supplies more varied candidates for later comparison, while helpfulness examples preserve legitimate engagement.

Why does supervised CAI include helpfulness examples?

They preserve instruction-following and legitimate engagement so the policy does not learn safety as silence or blanket refusal.

Are a principle, critique, revision, and supervised policy interchangeable?

No. The principle steers attention, the critique assesses, the revision supplies a candidate response, and supervised training turns examples into a response distribution.

Previous11 minConstitutional AI: The Critique–Revision Loop

Next11 minConstitutional AI: How RLAIF Ranks Answers

Constitutional AI: How Supervised Training Changes Behavior

The First Loop: What Supervised CAI Actually Changes

A single pipeline contains four different objects

FAQ