DeepSeek-R1: Rewards and Reasoning

Overview

This series explains how DeepSeek-R1 and R1-Zero turn outcome feedback into longer, more effective solution behavior: capable bases generate alternatives, verifiers score visible outcomes, and GRPO applies bounded relative pressure. It follows the move from pure RL to SFT, preference and safety signals, then tests benchmark gains, inference costs, distillation, reward hacking, and deployment limits. The evidence supports conditional capability gains, not human-like thought or universal transfer.

Learning outcomes

DeepSeek-R1’s reasoning gains are strongest where outcomes can be checked, especially in math, code, STEM, and logic.
GRPO turns within-problem comparisons into bounded policy updates; it does not define whether the reward captures the intended capability.
R1 needs cold-start data, SFT, preference, and safety signals because pure outcome RL does not cover readability or open-ended assistance.
Longer reasoning and distillation transfer selected behavior, but add compute costs and do not prove faithful thought or universal transfer.

Keywords

DeepSeek-R1GRPOVerifiersReasoningReinforcement learningSFTDistillationReward hackingInferenceSafety

Chapters

DeepSeek-R1 Feedback Contract

What outcome checks reveal—and leave hidden

11 min · What outcome checks reveal—and leave hidden

11 min

DeepSeek-R1: How GRPO Updates Policy

Why group scores need clipping and a reference policy

10 min · Why group scores need clipping and a reference policy

10 min

DeepSeek-R1: What Verifiers Can See

Why task design sets the reward boundary

11 min · Why task design sets the reward boundary

11 min

DeepSeek-R1: From Scores to Habits

What the training curve supports—and cannot prove

12 min · What the training curve supports—and cannot prove

12 min

DeepSeek-R1: Beyond Pure RL

Why one feedback source cannot cover the whole assistant

12 min · Why one feedback source cannot cover the whole assistant

12 min

DeepSeek-R1: What Its Evidence Shows

Benchmarks, adaptive compute, and smaller models

10 min · Benchmarks, adaptive compute, and smaller models

10 min

DeepSeek-R1: Where Rewards Break

Reward hacking, missing tools, and safety gaps

11 min · Reward hacking, missing tools, and safety gaps

11 min

DeepSeek-R1: Optimize the Feedback

What verifiers can select—and what they cannot

13 min · What verifiers can select—and what they cannot

13 min

Preview every chapter · Unlock full text, audio, and video from chapter 4

Plans

Ch.1-3 FreeCh.4-8 Premium

FAQ

How does DeepSeek-R1 learn reasoning without human-written steps?

DeepSeek-R1-Zero samples solutions from a capable base model, checks their outcomes, and uses GRPO to favor relatively successful outputs; later R1 adds supervised and preference-based training.

What can DeepSeek-R1’s reward system actually measure?

It can measure selected endpoints and outer properties such as answer correctness, test results, format, language consistency, preference, and safety, but not automatically every intermediate step or a faithful internal process.

Why does DeepSeek-R1 use both reinforcement learning and supervised fine-tuning?

DeepSeek-R1 uses reinforcement learning to explore verifiable outcomes and supervised fine-tuning to supply readable, general, and open-ended behaviors that its verifiers cannot reliably select.

What are the main limits of DeepSeek-R1’s reasoning gains?

The gains depend on base-model capacity, representative tasks, reliable evaluators, compute, and deployment controls; reward hacking, overthinking, missing tools, uneven transfer, and unsafe use remain possible.