InstructGPT and the Limits of Human Feedback

Overview

This series explains how InstructGPT turns human judgments into changed model behavior—and why that process is narrower than human intent. It follows the pipeline from labeler and customer prompts to supervised demonstrations, preference comparisons, a reward model, and PPO constrained by KL and pretraining signals. It then examines reported gains in instruction control and truthfulness, conditional toxicity improvements, unchanged selected bias measures, capability regressions and partial recovery through PPO-ptx, plus failures on false premises, excessive hedging, and multiple constraints. The evidence supports a powerful but bounded conclusion: feedback can improve a model on tested distributions, while its reward proxy, reference group, coverage, and deployment choices determine what the system captures and who remains outside the objective.

Learning outcomes

A 1.3B InstructGPT model was preferred to a 175B GPT-3 baseline in the reported API comparison, but this is not universal.
Human feedback becomes an optimization target only after prompts, demonstrations, comparisons, a reward model, and policy updates translate it into a score.
Reported gains are uneven: control and selected truthfulness improved, toxicity gains were prompt-dependent, and selected bias measures did not improve.
KL penalties and PPO-ptx mitigate policy drift and capability loss, but they do not make the reward proxy complete or universally safe.

Keywords

InstructGPTHuman feedbackAlignmentInstruction followingReward modelPrompt distributionSFTPPOPPO-ptxAlignment tax

Chapters

InstructGPT: Prediction vs User Intent

Why human feedback changes behavior without defining intent

9 min · Why human feedback changes behavior without defining intent

9 min

InstructGPT: Prompt Data and SFT

How demonstrations make instruction following learnable

11 min · How demonstrations make instruction following learnable

11 min

InstructGPT: Preference Rankings and Reward Models

How pairwise comparisons become an optimizable score

12 min · How pairwise comparisons become an optimizable score

12 min

InstructGPT: PPO, KL, and the Alignment Tax

How reward optimization trades off capability preservation

9 min · How reward optimization trades off capability preservation

9 min

InstructGPT: Control, Truthfulness, and Safety

What the paper’s evaluations actually establish

10 min · What the paper’s evaluations actually establish

10 min

InstructGPT: Transfer and Failure Modes

Why policy generalization remains partial and uneven

10 min · Why policy generalization remains partial and uneven

10 min

InstructGPT: Whose Intent Is Optimized?

What human feedback captures—and what it leaves out

11 min · What human feedback captures—and what it leaves out

11 min

Preview every chapter · Unlock full text, audio, and video from chapter 4

Plans

Ch.1-3 FreeCh.4-7 Premium

FAQ

What did InstructGPT improve over GPT-3?

InstructGPT improved instruction following and human preference on the study’s tested prompt distributions, with lower closed-domain hallucination and better TruthfulQA results; the gains were not universal.

How does InstructGPT turn human feedback into model behavior?

InstructGPT turns human feedback into behavior through selected prompts, demonstrations, ranked comparisons, a reward model, and PPO updates constrained by KL and, in PPO-ptx, pretraining gradients.

Does InstructGPT solve alignment?

No, InstructGPT does not solve alignment. It demonstrates substantial, distribution-specific behavioral gains, while leaving proxy limits, capability trade-offs, conditional safety, and representation questions unresolved.

Whose preferences does InstructGPT represent?

InstructGPT represents preferences captured by its prompt distribution, labelers, rubrics, comparison process, and evaluation setting—not an abstract universal humanity. Held-out labelers support repeatability within a related process, not universal representation.