Where the Constitution Stops

Constitutional AI’s most revealing evidence appears where its reward signal stops tracking the quality people intended. This chapter diagnoses that boundary in two directions: technically, when revisions and reinforcement learning improve a measurable proxy while degrading context-sensitive behavior; institutionally, when the constitution reflects only some of the people who define or experience the system’s consequences. The result is a way to tell a genuine limitation of CAI from a limitation of the specific evidence, and to see why the method belongs inside a broader safety and governance system.

When optimization outruns the target

The first symptom is a familiar one in learned systems: one score rises while another property that matters becomes weaker. In the supervised CAI experiments, harmlessness and combined helpfulness–harmlessness preference-model scores increased across repeated revisions, while pure helpfulness scores fell. That pattern does not mean the revision process failed. It means that “safer” and “more helpful” are not a single axis, and that repeated pressure on one axis can move the response away from another.

The causal chain is straightforward. A principle tells the model to remove harmful content. A critique identifies problems, a revision rewrites the answer, and the revised behavior enters supervised training. If the evaluator rewards caution more consistently than useful engagement, the model has an incentive to shorten, refuse, or surround an answer with warnings. The resulting sample may be less likely to provide dangerous assistance, yet also less able to address a legitimate question. The optimization has followed the observable target; the target was narrower than the full quality of the interaction.

The same problem can appear in the reinforcement-learning stage with more leverage. A preference model assigns a scalar reward to responses, and PPO repeatedly adjusts the policy toward higher scores. Any shortcut in the preference model can therefore be repeated across many policy updates. The shortcut need not be malicious or obviously wrong. It may be a phrase associated with care, a rigid refusal style, or an exaggerated condemnation that often happened to win the comparisons. Once the policy learns that surface pattern, the reward can rise without a proportional improvement in context-sensitive judgment.

The CAI paper reports this directly as over-training or Goodharting. RL-CAI models can become overly harsh when responding to harmful prompts, or insert boilerplate reassurance into many red-team conversations. The repeated language is not necessarily harmful in every instance. The issue is that a response pattern originally useful in some contexts becomes a general-purpose way to satisfy the reward. A model may sound supportive while failing to answer the actual question, or may condemn a request so strongly that it misreads a benign discussion.

This is not a flaw unique to constitutions. Any learned reward can be optimized beyond the region where it is a good proxy. CAI makes the failure easier to see because the original principle is written in ordinary language, inviting a comparison between the rule and the behavior. The presence of a visible rule does not prevent the preference model from learning a shortcut. It only gives people a better starting point for asking where the shortcut entered.

There are several possible locations. The principle may not distinguish discussing a harmful topic from assisting a harmful act. The prompt may not provide enough context to infer the user’s intent.