Overview
Scaling laws describe how loss changes with model size, training tokens, and compute in dense autoregressive Transformers. This series compares Kaplan's model-heavy frontier with Chinchilla's approximately balanced one, then turns the evidence into a bottleneck-first allocation decision. The result is conditional: optimize for the real objective, check data and schedules, and validate task, deployment, and risk outcomes before extrapolating.
Learning outcomes
- At fixed training compute, model size and training tokens are coupled allocation choices, not interchangeable resources.
- Kaplan's frontier favored larger models in its tested dense-Transformer regime, but its finite-data analysis warned of a later bottleneck.
- Chinchilla's experiments shifted the prior toward approximately balanced growth of parameters and tokens, not a fixed ratio.
- Scaling laws guide local experiments; task performance, serving cost, data quality, overlap, and safety still require separate validation.
Keywords
Preview every chapter · Unlock full text, audio, and video from chapter 4
PlansFAQ
What do scaling laws for language models actually measure?
They empirically describe how measured cross-entropy loss changes with model size, training tokens, and compute for tested dense autoregressive Transformers. They do not directly measure intelligence or deployment safety.
Why is compute allocation more important than parameter count alone?
Because a fixed budget must be divided among capacity, token exposure, and optimization, so extra parameters can be underused when data or training time is limiting.
What did the Chinchilla result show?
It showed that, at similar training FLOPs, a 70B Chinchilla trained on 1.4T tokens outperformed 280B Gopher trained on roughly 300B tokens across many reported evaluations. The complete setups differed, so tokens alone cannot be credited for the entire gain.
How should a team choose its next scaling step?
A team should start with its objective, diagnose whether capacity, data, or optimization is limiting, match the horizon and schedule, and validate loss, task outcomes, overlap, and safety before scaling.