Overview
Constitutional AI turns written principles into model-generated critiques, revisions, comparisons, preference-model rewards, and policy updates. This series follows both loops from the RLHF architecture, showing how they can reduce direct human harmlessness labeling and support less harmful, less evasive behavior in reported tests. It also explains the limits: proxies can be miscalibrated or optimized into boilerplate and overreaction, while human choices, independent evaluation, deployment boundaries, and governance remain necessary.
Learning outcomes
- CAI can turn written principles into training feedback, but only through model-generated and learned proxies.
- The supervised critique–revision loop can create a safer, more varied starting policy without proving robust harmlessness.
- RLAIF can improve tested harmlessness and reduce evasiveness, while calibration and distribution shift remain constraints.
- Human choices still define the target, evaluate failures, set deployment boundaries, and decide when evidence is insufficient.
Keywords
Preview every chapter · Unlock full text, audio, and video from chapter 4
PlansFAQ
What does Constitutional AI change compared with RLHF?
It replaces direct human harmlessness comparisons with AI-generated, principle-guided comparisons, while retaining human helpfulness feedback and the broader reward-learning architecture.
How does Constitutional AI turn principles into model behavior?
It uses principles to prompt critiques and revisions for supervised training, then to compare response pairs whose judgments train a preference model used by PPO.
Does Constitutional AI make language models safe or value-neutral?
No; Constitutional AI is neither a guarantee of safety nor a value-neutral process. Its reported gains are conditional on selected prompts, evaluators, model snapshots, and reward signals, while human choices still define the target.
Why can Constitutional AI become overconfident or evasive?
Because a narrow or miscalibrated reward can be amplified by optimization, producing overreaction or boilerplate even while measured scores rise.