Overview

Qwen3's benchmark results are conditional measurements defined by model, mode, sampling, output limit, and task. The flagship is reported to beat DeepSeek-R1 on 17 of 23 thinking benchmarks and GPT-4o on 18 of 23 non-thinking benchmarks, but those counts do not mean dominance on every task. Smaller-model, multilingual, and long-context results support portability and reach with important qualifications, including lower listed thinking-mode RULER scores. The report supports in-environment testing, not universal latency, cost, safety, or production-reliability claims.

Learning outcomes

Qwen3 benchmark scores belong to specific mode, sampling, output, and task protocols rather than a single capability number.
Qwen3-235B-A22B is reported above DeepSeek-R1 on 17 of 23 thinking benchmarks and GPT-4o on 18 of 23 non-thinking benchmarks.
Long-context RULER results favor non-thinking mode for the listed Qwen3 models, showing that more reasoning is not always better.

Keywords

Qwen3BenchmarksEvaluationSamplingRULERMultilingual

What the Benchmarks Actually Show—and What They Do Not

Qwen3's benchmark tables are meant to support a family-level claim: one open-weight line can combine deliberate reasoning, direct responses, and useful performance across sizes and languages. This chapter tests that claim under the report's own evaluation conditions. The reader will see why thinking and non-thinking scores must remain separate, how the flagship's “17 of 23” and “18 of 23” comparisons translate into task-level evidence, and why smaller-model, multilingual, and long-context results turn a leaderboard story into a conditional judgment.

Read the operating point before reading the score

A benchmark number is the result of a model, a prompt, a sampling policy, an output limit, and a task definition acting together. Qwen3's report evaluates thinking and non-thinking modes separately. For thinking mode it uses temperature 0.6, top-p 0.95, and top-k 20. For non-thinking mode it uses temperature 0.7, top-p 0.8, and top-k 20. The maximum output is 32,768 tokens, extended to 38,912 for AIME'24 and AIME'25. Those settings are not peripheral implementation notes: they define the operating point at which the reported capability was measured.

The report also uses different evaluation procedures for different tasks. The suite includes general knowledge and alignment, mathematics and text reasoning, coding and agent tasks, and multilingual instruction, knowledge, math, and logic. AIME problems are sampled 64 times and averaged. Codeforces problems can receive up to eight independent reasoning attempts. Multilingual benchmarks are sampled differently and cover different language sets. A result is therefore not a single property called “Qwen3 intelligence.” It is a measurement of a particular behavior under a particular protocol.

flowchart LR A["Benchmark claim"] --> B{"Operating mode"} B -->|"Thinking"| C["Model size and active computation"] B -->|"Non-thinking"| C C --> D["Sampling and output protocol"] D --> E["Task-specific result"] E --> F["Applicability to a workload"] F --> G["In-environment validation"] G --> H["Boundary: not a universal ranking"]

This reading order prevents two common mistakes. The first is to compare thinking and non-thinking scores as if they were produced by the same procedure and then attribute every difference to the presence of reasoning alone. The second is to count wins across a table without asking whether the tasks have equal importance, whether baselines used matching prompts and sampling, or whether a benchmark measures the capability a deployment actually needs.

The flagship thinking table is nevertheless substantial evidence. Qwen3-235B-A22B is compared with OpenAI-o1, DeepSeek-R1, Grok-3-Beta in a thinking setting, and Gemini 2.5 Pro. The report gives Qwen3 scores of 85.7 on AIME'24, 81.5 on AIME'25, 70.7 on BFCL v3, and 70.7 on LiveCodeBench v5. Its Codeforces rating is listed as 2056, at the 98.2 percentile. The authors count Qwen3 as outperforming DeepSeek-R1 on 17 of the 23 tabled benchmarks while using 60% of DeepSeek-R1's activated parameters and 35% of its total parameters.

Published by DeepMox

FAQ

Why must Qwen3 thinking and non-thinking scores be read separately?

Qwen3's thinking and non-thinking scores use different sampling settings and intended output behaviors, so their difference is not a clean causal comparison of reasoning alone. Each score describes its own operating point and task protocol.

What do Qwen3's 17-of-23 and 18-of-23 comparisons mean?

They mean the authors counted Qwen3-235B-A22B above DeepSeek-R1 on 17 of 23 listed thinking benchmarks and above GPT-4o on 18 of 23 listed non-thinking benchmarks under their protocols. They do not mean universal dominance.

Did Qwen3's long-context tests favor thinking mode?

No. For the listed RULER models, Qwen3's thinking-mode scores were lower than the corresponding non-thinking scores; the authors used an 8,192-token budget and treated interference with retrieval as a hypothesis.

Previous10 minQwen3 Distillation: Reasoning at Smaller Sizes

Next12 minQwen3: Reasoning as a Controllable Resource

Qwen3 Benchmarks: What the Scores Show

What the Benchmarks Actually Show—and What They Do Not

Read the operating point before reading the score

FAQ