Overview

At approximately matched training FLOPs, 70B Chinchilla trained on 1.4T tokens outperformed 280B Gopher trained on roughly 300B tokens across many reported evaluations. This chapter explains the broad but non-uniform task gains, lower serving cost, leakage and data risks, and why the full setups prevent a tokens-alone causal claim.

Learning outcomes

At similar training FLOPs, 70B Chinchilla trained on 1.4T tokens outperformed 280B Gopher trained on roughly 300B tokens across many reported evaluations.
The gains were broad but not uniform: MMLU improved on 51 of 57 subjects, with four subjects performing worse.
The result supports a full-setup allocation advantage, not a tokens-alone causal claim; lower serving cost is an additional conditional benefit.

Keywords

Scaling lawsChinchillaGopherBenchmarksInferenceContamination

The 70B Model That Beat the 280B Model

The Chinchilla study's most persuasive contribution is not its slogan about token counts; it is a matched-budget case. Hoffmann and colleagues trained a 70-billion-parameter model on 1.4 trillion tokens and compared it with Gopher, a 280-billion-parameter model trained on roughly 300 billion tokens, at approximately the same training FLOPs. This chapter examines what that comparison establishes, how the results vary across evaluations, and why the smaller model's inference cost and data risks belong in the same decision as its pre-training loss.

A matched budget makes the allocation visible

The comparison tests the frontier predicted in the previous chapter. If model size and training tokens should grow at roughly equal rates with compute, then the compute used for Gopher should not automatically be spent on another model of similar size. It should support a smaller model that sees far more tokens. Chinchilla is four times smaller than Gopher in parameter count and is trained on about four times as many tokens. The point is not that 70 billion is intrinsically a better size than 280 billion. The point is that the two sizes are paired with different training exposures under a common budget.

That makes Chinchilla a frontier test rather than simply a smaller-model experiment. A smaller model trained on the same 300 billion tokens would answer a different question: how much capacity can be removed before performance falls? Chinchilla instead changes capacity and data together so that the total training computation remains comparable. The result can therefore speak to allocation: whether the loss and measured capabilities gained by more tokens compensate for the capacity given up.

The paper predicts that the Gopher budget has an optimal model somewhere between roughly 40 and 70 billion parameters, then tests a 70-billion-parameter model called Chinchilla on 1.4 trillion tokens. Both models use the same broad architecture and training setup, with specified differences in optimizer, tokenizer, and precision handling. The result is a direct but not perfect test of the frontier. It is direct because the model/data choice follows the prior estimate and the training FLOPs are matched. It is not perfect because the complete training setups are not identical in every variable.

The first outcome is language modeling. Chinchilla significantly outperforms Gopher across the reported evaluation subsets of The Pile. On WikiText-103, it achieves a perplexity of 7.16 compared with Gopher's 7.75. The paper cautions that the comparison is vulnerable to train/test leakage: Chinchilla sees four times more data, so its larger exposure may increase overlap with some evaluation material. That caution does not erase the result, but it changes how strongly the result should be used. On language-modeling evaluations, the direction is clear; the clean separation between better generalization and more exposure to evaluation-like text is less clear.

Published by DeepMox

FAQ

How broad were Chinchilla's gains over Gopher?

They were broad but not universal: Chinchilla led on 51 of 57 MMLU subjects and all but four of 62 BIG-bench tasks, with some regressions.

What does the Chinchilla-Gopher comparison fail to isolate?

It does not isolate token count because the models also differed in optimizer, tokenizer, and precision handling, while greater data exposure raises leakage concerns.

Why can a smaller model be better for deployment?

A smaller model can be better when it retains the required task capability, because it needs less memory and inference computation for repeated serving or fine-tuning. The advantage depends on the use pattern.

Previous9 minScaling Laws: Why Chinchilla Moved the Frontier

Next11 minScaling Laws: How to Choose the Next FLOP

Scaling Laws: Chinchilla vs Gopher

The 70B Model That Beat the 280B Model

A matched budget makes the allocation visible

FAQ