Overview

DeepSeek-R1’s reported scores measure different capabilities and feedback relationships, not one universal reasoning score. Gains are strongest on several reasoning-heavy benchmarks, while general preference results reflect later SFT and mixed RL. At inference, R1 allocates more tokens to harder problems and benefits from multiple samples, but cost and overthinking remain. Distillation transfers selected teacher behavior to smaller models without reproducing the discovery process; all claims remain protocol- and evaluation-dependent.

Learning outcomes

R1’s benchmark scores are not interchangeable: answer checks, executable tests, preference judgments, and repository tasks measure different relationships to training.
Adaptive reasoning allocates fewer tokens to easier problems and more to harder ones, but long generation increases cost and can overthink.
Distillation transfers curated teacher behavior to smaller models economically, not the teacher’s full search process or capability ceiling.

Keywords

DeepSeek-R1BenchmarksInferenceDistillationTransferCompute

What the Evidence Actually Shows

DeepSeek-R1’s benchmark results are evidence about several different capabilities, not one universal “reasoning score.” This chapter interprets the reported pattern: which gains line up most closely with verifiable reward design, why general preference results move on a different track, how R1 spends computation at inference time, and what distillation preserves in smaller models. The goal is to read the evidence as a set of conditional measurements rather than as a leaderboard verdict.

The Gains Concentrate Where the Feedback Is Concrete

The most direct evidence for the reward-centered thesis appears in mathematics, algorithmic coding, STEM, and long-context reasoning. In the paper’s comparison, DeepSeek-R1 reaches 79.8% on AIME 2024 pass@1, 97.3% on MATH-500, a 96.3 percentile on Codeforces, 65.9% on LiveCodeBench, 84.0% on MMLU-Pro, and 82.5% on FRAMES. It also reports 49.2% resolved on SWE-Bench Verified and 87.6% on the length-controlled AlpacaEval 2.0 win rate.

Those numbers are not interchangeable. AIME and MATH-500 test mathematical problem solving with answer-based evaluation. Codeforces and LiveCodeBench focus on algorithmic programming. MMLU-Pro spans knowledge and reasoning across subjects. FRAMES tests answering over long documents. SWE-Bench Verified concerns software-engineering issues, where solving a real repository task is slower and harder to evaluate than producing a competition solution. AlpacaEval is a preference comparison for open-ended responses. The fact that R1 performs well across them is notable, but the mechanisms behind the scores are different.

The comparison with DeepSeek-V3 makes the pattern clearer. R1’s largest relative gains appear in competitive programming, mathematics, STEM-heavy tests, and long-context analysis—the areas where additional reasoning trajectories and outcome checks can plausibly help. R1 also improves on preference-oriented evaluations after its later SFT and RL stages. DeepSeek-V3 remains stronger on some instruction-following dimensions in the detailed table, which is consistent with the idea that the two models were optimized toward different behavior profiles.

This is why the reward design is a better explanatory lens than the raw ranking. R1-Zero’s rule rewards directly target correct math answers, executable code, STEM choices, and logic outcomes. R1 adds general data and preference rewards later. The resulting score pattern is not proof that every improvement came from pure reasoning RL, but it is consistent with a staged system whose objectives leave different signatures.

The domain breakdown also matters. On a 2024 collection of competition problems, R1 is reported to be relatively strong in number theory and algebra, while geometry and combinatorics remain weaker. A model can be excellent at one class of verifiable problems and still have uneven coverage within the same broad field. A reward contract selects what it can see; it does not guarantee uniform competence across all forms of mathematics or code.

A useful way to read the table is to ask how close each metric is to the training feedback. A correct answer on AIME or MATH-500 resembles the kind of endpoint that answer matching can reward. A Codeforces result resembles the executable feedback available during code training, although the evaluation set and protocol are not identical to the training environment. An AlpacaEval win rate instead reflects a later preference contract, while SWE-Bench Verified introduces long evaluation cycles and real-world engineering context that the paper says received less extensive large-scale RL. The scores can all be high without being evidence of one common mechanism.

The comparison with DeepSeek-V3 is especially informative because the two models share the same base checkpoint but receive different post-training emphasis.

Published by DeepMox

FAQ

Why can’t DeepSeek-R1’s benchmark scores be treated as one reasoning score?

They cannot be treated as one score because the benchmarks use different tasks, evaluators, protocols, and post-training relationships.

How does DeepSeek-R1 allocate inference-time computation?

It uses fewer than 7,000 thinking tokens on easier competition problems and more than 18,000 on the hardest in the reported analysis, trading adaptive effort for cost.

What does DeepSeek-R1 distillation transfer to smaller models?

It transfers selected teacher outputs as supervised training data, not the teacher’s entire search history or internal reasoning dynamics.

Previous12 minDeepSeek-R1: Beyond Pure RL

Next11 minDeepSeek-R1: Where Rewards Break

DeepSeek-R1: What Its Evidence Shows

What the Evidence Actually Shows

The Gains Concentrate Where the Feedback Is Concrete

FAQ