A Better Way to Spend the Next FLOP

The subject of this series has been the compute allocation problem for dense autoregressive Transformer language models: how to divide a finite training budget between model capacity, training tokens, and optimization time. The answer is not a permanent parameter-to-token ratio. It is a conditional judgment about which resource is limiting, which outcome matters, and how closely the planned run resembles the evidence behind the chosen frontier. This conclusion brings the two scaling studies together into a decision model that can guide a new training plan without pretending that empirical exponents are laws of nature.

One constrained question, two different frontiers

Kaplan et al. and Chinchilla ask the same broad question: for a given amount of compute, which combination of model size and training process gives the best measured loss? Their answers differ because the frontier is inferred from a training system, not read directly from a parameter count. The studies use different model ranges, data regimes, schedule choices, compute conventions, and fitting methods. Those differences are not noise around a single number; they determine what each experiment can see.

The first study's framework emphasized sample efficiency. Larger models reached a target loss with fewer minimum steps, so after correcting for batch-size inefficiency its empirical frontier put most additional compute into parameters. The summary allocation was approximately Nopt ∝ Cmin^0.73, with data and serial steps growing much more slowly. That result was coherent for the observed dense-Transformer regime and the objective of minimizing compute, but it was never a claim that data would remain unimportant at every future scale. Its own finite-data analysis predicted a tension: the data needed to keep larger models from becoming data-limited would eventually grow faster than the compute-efficient plan could productively consume it.

Chinchilla remeasured the frontier with hundreds of models, more training horizons, schedule lengths matched to those horizons, and three complementary fitting approaches. Those methods converged on approximately equal exponents for model size and training tokens: roughly 0.46–0.50 for parameters and 0.50–0.54 for tokens. The direct Chinchilla/Gopher comparison then tested that direction at a large budget: a 70-billion-parameter model trained on 1.4 trillion tokens outperformed the 280-billion-parameter Gopher on many reported evaluations at similar training FLOPs.

The right synthesis is not that the earlier paper was simply wrong or that the later paper discovered a universal replacement. The stronger, more useful conclusion is that frontier estimates are conditional and revisable. If a training setup resembles Chinchilla's dense-model, schedule-matched, data-expanding regime, a roughly balanced growth of model size and tokens is a better starting prior than a parameter-heavy allocation. If the objective is different, the data regime is different, or the architecture changes the relationship between capacity and FLOPs, the prior must be tested again.

This is why a scaling law should be treated like a map with a stated coordinate system. The axes need definitions: whether parameters include embeddings, how FLOPs are counted, what a token represents, and whether repeated data exposure is part of the regime. The destination also needs a definition: minimum pre-training loss, minimum elapsed time, downstream task performance, inference cost, or some combination. A frontier can be optimal for one destination and poor for another.

Seen this way, the two studies are not competing answers to two different questions. They are different measurements of the same question under different experimental visibility. Kaplan's frontier made the value of sample efficiency and large models explicit. Chinchilla made the cost of insufficient training exposure explicit.