Overview

RLAIF turns a constitutional principle into a local comparison: a feedback model receives a prompt, two candidate responses, and the principle, then produces choice probabilities. Those soft labels train a hybrid preference model combining AI-generated harmlessness comparisons with human helpfulness data; PPO uses its scores to update the policy. Using the SL-CAI policy for both data generation and initialization reduces early distribution mismatch, while 16-principle ensembling and 40–60 percent probability clamping improve reported robustness. The chain remains a probabilistic reward proxy, not the constitution itself.

Learning outcomes

RLAIF compares two responses under a supplied principle; it does not assign an absolute moral score.
Soft choice probabilities preserve uncertainty, but chain-of-thought can make them extreme and require calibration.
A hybrid preference model scales AI harmlessness feedback with human helpfulness judgments, while PPO can amplify proxy errors.

Keywords

Constitutional AIRLAIFpreference modelsoft labelscalibrationPPO

From Principles to Probabilistic Preferences

RLAIF is the point in Constitutional AI where a principle stops merely asking for a better answer and starts selecting among answers. This chapter traces that comparison interface and the path from a model’s choice to a preference-model reward and an RL policy update. The distinction matters because four different objects are easy to collapse into one: the constitutional rule, the feedback model’s judgment, the preference model’s score, and the reward that changes the policy.

The comparison interface has more parts than a verdict

The process begins with a prompt and a policy capable of producing responses. For each prompt, the policy generates a pair of candidate answers. The pair is then presented to a feedback model along with a constitutional principle. Instead of asking the feedback model to write a free-form evaluation with no defined output, the system turns the task into a multiple-choice question: which response is more helpful, honest, or harmless under this principle?

That formatting decision is more consequential than it may appear. The feedback model is not asked to declare an absolute score for either answer. It is asked to choose between two alternatives in a fixed interface. The conversation provides context, the principle defines the criterion, and the response pair defines the local decision. The answer is then read from the model’s probabilities for the choice tokens, rather than from a separate human label.

A simple example can make the boundary clear without relying on a dangerous request. Suppose a user asks a sensitive question and one candidate answer gives a terse refusal while the other explains the concern and offers a safe way to continue. A principle that values harmlessness without excessive reactivity may favor the second response. A principle focused on avoiding illegal assistance might favor the response that draws a firmer boundary. The feedback model’s job is not to invent a new constitution in either case. It interprets the supplied principle in the context of the two candidates.

The research uses multiple principles and samples one for each comparison. The principles ask for combinations such as helpfulness, honesty, harmlessness, politeness, and ethical awareness, while warning against toxicity, social bias, illegal assistance, preachiness, or overreaction. Sampling changes the criterion’s wording and emphasis across examples. It is a way to expose the preference model to several constitutional views of the same broad goal rather than teaching it one rigid sentence.

The output of the feedback model is not just a hard winner. The authors compute the log probabilities assigned to the two answer choices and normalize them into preference targets. If the model assigns similar probabilities to A and B, the training example retains some indication that the comparison is uncertain. If it strongly favors one choice, the target is more decisive. This is the role of a soft label: it carries the degree of preference instead of erasing it into a binary zero or one.

That distinction changes the information passed to the next component.

Published by DeepMox

FAQ

What does an RLAIF feedback model compare?

It compares two candidate responses to the same prompt under a supplied constitutional principle, choosing which is more helpful, honest, or harmless.

Why use soft labels instead of a hard winner?

Soft labels retain how separated the feedback model’s choice probabilities are, so close comparisons need not look as decisive as obvious ones.

Why are chain-of-thought probabilities clamped in CAI?

The main chain-of-thought RL-CAI setup clamps them to 40–60 percent because reasoning improved choices but often pushed probabilities toward extreme values.

Previous10 minConstitutional AI: How Supervised Training Changes Behavior

Next12 minConstitutional AI: Testing Reward Calibration

Constitutional AI: How RLAIF Ranks Answers

From Principles to Probabilistic Preferences

The comparison interface has more parts than a verdict

FAQ