Mitigation Is a System, Not a Refusal
GPT-4’s safety behavior is the result of several controls acting at different points in its lifecycle and deployment environment. This chapter examines how the report moves from data filtering and human-feedback training to classifiers, policies, monitoring, and product design. The practical judgment is to match each control to the failure surface it can actually affect: a refusal can change a response, but it cannot erase latent capability, settle a context-dependent question, or replace oversight around a tool-enabled system.
The first boundary: reducing exposure before training
The mitigation pipeline begins before the model produces a response. OpenAI says that it filtered the GPT-4 pretraining mix to reduce inappropriate erotic text, using internally trained classifiers and a lexicon-based method to identify documents with a high likelihood of containing that material. Those documents were removed from the pretraining set.
This kind of intervention changes the data available to the training process. It can reduce exposure to selected patterns and lower the chance that they are reproduced, but it is not a complete behavioral specification. A filter is built around a category definition, a classifier, and a threshold; it can miss material, remove material that is not actually harmful, or leave other risk categories untouched. Removing a class of documents also does not remove the model’s broader ability to generate language learned from the remaining data.
The distinction is useful because pretraining and post-training answer different problems. Data filtering attempts to change what enters the learning environment. Post-training begins with a model that already has broad learned behavior and tries to steer how it responds. Neither layer owns the entire safety state. A model can be safer on a selected content category while remaining unreliable, biased, or vulnerable to a different prompt.
How human feedback steers behavior
After pretraining, the primary method for shaping GPT-4-launch behavior was RLHF. The process described in the System Card has three linked stages. Human trainers provide demonstrations of how a model should respond to an input. Supervised fine-tuning trains a model to imitate those demonstrations. Trainers then rank several outputs for a given input; those rankings train a reward model that predicts the average preference expressed by the labelers. The system uses that reward signal to fine-tune the supervised model with reinforcement learning, specifically PPO.
The stages create a chain of ownership. Demonstrations define examples of desired behavior. The reward model compresses rankings into a signal that can be applied across outputs. PPO uses that signal to change the policy that generates responses. If the demonstrations or ranking instructions are underspecified, the resulting reward can favor behavior that looks acceptable in one case but fails at an edge case. The report says that RLHF makes the models significantly safer while also noting brittleness, excessive caution, and over-refusal after the process.
The goal is not simply to maximize refusal. Trainers are instructed to reward refusals for certain harmful requests and appropriate responses to sensitive prompts, while also rewarding non-refusal on prompts known to be safe and answerable. This creates a genuine control problem. A model that complies too readily can produce harmful content; a model that refuses too broadly can deny benign assistance, create unequal quality of service, or teach users to disregard its safety signals. The target is context-sensitive behavior, but the training signal is necessarily an approximation of that target.
The report adds rule-based reward models, or RBRMs, to make the steering more fine-grained. An RBRM is a zero-shot GPT-4 classifier given an optional prompt, the policy model’s output, and a human-written rubric describing how the output should be judged.