Overview

DeepSeek-R1 turns R1-Zero’s raw outcome-RL discovery into a broader assistant through readable cold-start examples, a language-aware first RL stage, rejection sampling, mixed reasoning and non-reasoning SFT, and a final RL stage with preference and safety rewards. The stages solve different observability problems and trade gains across capabilities; internal checkpoints support a pipeline explanation, not an isolated causal percentage for each component.

Learning outcomes

Cold-start examples make R1-Zero’s difficult-to-read behavior a more usable starting point, without proving human-like narration.
Rejection sampling packages selected, correct, readable trajectories as reusable supervised data rather than copying every RL output.
Mixed SFT and later preference, safety, format, and language rewards broaden R1, with trade-offs and no single stage owning all gains.

Keywords

DeepSeek-R1SFTRejection samplingPreferenceSafetyReadability

Why DeepSeek-R1 Needs More Than Pure RL

DeepSeek-R1-Zero shows what unconstrained outcome-based reinforcement learning can discover; DeepSeek-R1 shows why discovery is only one stage of building a usable assistant. This chapter follows the transitions between those stages: readable cold-start data, a first RL pass, rejection sampling, mixed supervised fine-tuning, and a second RL pass with preference and safety signals. The point is not to recite a pipeline, but to see what each transition preserves, what it repairs, and why the final gains cannot be credited to one technique alone.

Stage One: Make Raw Discovery Usable

R1-Zero’s strength is also its first problem. It learns to spend more effort on difficult, verifiable questions, but the resulting responses can be hard to read. The paper reports poor readability and language mixing, including English and Chinese appearing in the same chain of thought. Its rule-based RL is also narrowly concentrated on reasoning tasks, so strong mathematical or coding performance does not automatically produce good writing or open-domain answers.

DeepSeek-R1 begins by adding a small cold-start collection of long reasoning examples. The examples are designed to be conversational and human-aligned: they begin by understanding the problem, use reflection and verification, maintain language consistency, and present the reasoning from a first-person perspective. Human annotators first reshape selected R1-Zero traces into a more natural style. An LLM then rewrites additional examples in a similar style, and human reviewers verify the generated data.

This transition does not mean that the authors decided human-like narration was proof of human-like intelligence. The paper explicitly warns that the vivid first-person patterns are engineered heuristics and may create unwarranted trust. The cold-start stage has a product purpose: it gives the next policy a readable initial behavior from which further optimization can proceed. It is a user-interface intervention as much as a reasoning intervention.

The first RL stage then tries to improve reasoning while retaining that conversational starting point. Its reward mix includes the rule-based accuracy and format signals used for reasoning, plus language consistency. The language reward measures the proportion of words in the target language. In the paper’s ablation, language consistency remains stable when the reward is applied, while coding performance declines slightly. This is a real trade-off, not an implementation accident: making the output easier for people to follow can move the policy away from a narrow optimum on one coding measure.

The stage therefore has two owners of behavior. Rule-based rewards continue to own whether a math, code, STEM, or logic outcome is correct. Language consistency owns whether the reasoning is expressed in the requested language. Neither signal alone defines a good assistant. The policy is being asked to preserve useful exploration while satisfying a more legible outer contract.

Stage Two: Turn Successful Exploration into Training Data

Reinforcement learning produces many candidate trajectories, but not all of them are suitable examples for supervised training. Some are wrong. Others reach the correct answer through repetition, language mixing, long paragraphs, or code blocks that make the result difficult to use. DeepSeek-R1’s next transition is rejection sampling: generate multiple responses for a prompt, retain correct and acceptable ones, and use the selected set as supervised data.

Published by DeepMox

FAQ

Why does DeepSeek-R1 add cold-start data after R1-Zero?

It adds cold-start data to give later training a readable, conversational starting behavior while preserving exploration; the engineered style is not evidence of human-like intelligence.

What does rejection sampling contribute to DeepSeek-R1?

It filters sampled trajectories for correctness and readability, converting selected RL behavior into reusable supervised fine-tuning examples.

Why did DeepSeek-R1’s first cold-start checkpoint lose AIME performance?

The first cold-start checkpoint fell on AIME 2024 while IF-Eval improved because the new response contract prioritized instruction following and readability differently from narrow reasoning performance.

Previous12 minDeepSeek-R1: From Scores to Habits

Next10 minDeepSeek-R1: What Its Evidence Shows

DeepSeek-R1: Beyond Pure RL

Why DeepSeek-R1 Needs More Than Pure RL

Stage One: Make Raw Discovery Usable

Stage Two: Turn Successful Exploration into Training Data

FAQ