Overview

Constitutional AI’s judging experiments test whether a model can compare assistant responses under a fixed interface, not whether it possesses general moral competence. Across 438 HHH comparisons and separate harm-identification tests, few-shot examples, chain-of-thought, and principle ensembles improve reported judgment, but chain-of-thought probabilities can become overconfident and are clamped to 40–60 percent in the main RL-CAI setup. RL-CAI improves tested harmlessness while retaining useful helpfulness under the reported protocol, yet over-training can produce harshness or boilerplate, and curated evaluations do not establish safety under arbitrary use.

Learning outcomes

Comparison accuracy under a test interface does not establish general moral competence or safe deployment.
Chain-of-thought and principle ensembles can improve judging, but confidence needs separate calibration.
The reported RL-CAI results form a rough helpfulness–harmlessness frontier; over-training can trade context-sensitive behavior for higher proxy scores.

Keywords

Constitutional AIcalibrationchain-of-thoughtharmlessnesshelpfulnessGoodharting

When the Reward Learns to Sound Certain

The central evidence question for Constitutional AI is not simply whether a model can choose an answer. It is whether the choice is accurate enough, calibrated enough, and broad enough to become a reward without producing new distortions. This chapter interprets the CAI experiments at those three levels: what the judging tests measure, why chain-of-thought and principle ensembles change the signal, and how the reported helpfulness, harmlessness, and non-evasiveness results should be read as a frontier rather than a single safety score.

What a judging model can actually demonstrate

The paper tests whether language models can identify a better assistant response in a multiple-choice format. This is a useful intermediate capability for RLAIF because the feedback model does not need to write a complete answer to the user. It needs to compare two answers under a stated criterion. The evaluation draws on earlier HHH comparisons and adds harder cases, especially cases where a thoughtful, harmless answer should beat an evasive refusal.

The combined set contains 438 binary comparisons intended to test helpfulness, honesty, and harmlessness. The researchers present a conversation, two candidate responses, and a question asking which response is preferable. They compare several ways of answering: a pretrained language model, a preference model trained on human feedback, a helpful RLHF policy, and versions prompted with few-shot examples or chain-of-thought reasoning. The task isolates judgment from generation. A model can be poor at writing an answer yet still recognize which of two answers is better, or it can write fluently while choosing badly.

That isolation makes the result informative but narrow. Success means the model’s choice agrees with the labels used to construct the evaluation. It does not establish that the model has a general moral theory, that it understands the consequences of an answer in the world, or that its judgment will transfer to an unseen deployment. The comparison itself supplies the alternatives and the criterion. A model that succeeds on the test has demonstrated competence under that interface.

The paper adds harm-focused tests that sharpen the distinction. One set contains 254 balanced conversations in which the task is to identify harmful versus ethical assistant behavior. Another contains 287 examples for classifying a harmful interaction into one of nine common labels. Few-shot prompting and chain-of-thought improve zero-shot performance on these tasks. That is evidence that format and reasoning prompts can help a model make the requested distinctions. It is not evidence that the categories exhaust the possible forms of harm or that the model will detect a subtle harm when no category has been supplied.

Few-shot examples help for a concrete reason. They show the feedback model what counts as a conversation, how the principle should be applied, which answer format is expected, and how much explanation belongs before the choice. Without that framing, the model may continue the conversation, answer the user directly, or produce a critique instead of a label. The examples narrow the interface’s ambiguity.

Published by DeepMox

FAQ

What do the CAI judging tests actually measure?

They measure whether a model chooses the labeled-preferred response under a particular comparison format, criterion, and evaluation set.

Why is RL-CAI described as a Pareto frontier?

Because the reported models trade off helpfulness and harmlessness rather than producing one model that dominates both objectives; the preferred point depends on their weighting.

What happens when RL-CAI is over-trained?

It can become overly harsh or repeat boilerplate reassurance, raising a proxy score without proportionally improving context-sensitive support.

Previous11 minConstitutional AI: How RLAIF Ranks Answers

Next10 minConstitutional AI: Where the Constitution Stops

Constitutional AI: Testing Reward Calibration

When the Reward Learns to Sound Certain

What a judging model can actually demonstrate

FAQ