The Scaling Problem Is an Allocation Problem
Scaling laws for neural language models are not primarily a story about how many parameters fit inside a model. The subject of this series is the measured scaling behavior of dense, autoregressive Transformer language models: how their performance changes as researchers vary model size, training data, and computation. This opening focuses on why those resources must be considered together, so readers can judge a practical question: given a fixed training budget, when is a larger model actually the better choice?
That question matters because training a large language model is an investment made before its results are known. A team may know how many accelerators it can use and for how long, but still have to choose whether to spend the budget on a model with more parameters, on more training tokens, or on a longer and more carefully scheduled run. These choices affect more than a training curve. They shape the model's eventual evaluation results, the amount of hardware needed to serve it, and how much training data and evaluation work the system requires.
The subject is the trade-off hidden inside “bigger”
A parameter count is an easy number to display, and the history of language models has often been told as a sequence of larger numbers. But parameter count is only one part of a training system. A model with more parameters can represent more complicated functions, yet it also consumes more computation every time it processes a token. If the training set or the compute budget does not grow in a compatible way, the extra capacity may not be used effectively.
A simple thought experiment shows why the choice is unavoidable. Suppose the amount of available computation is fixed. Making the model twice as large changes how much work is required for every training token. Unless the team also changes the batch size, the number of update steps, or the number of tokens processed, the larger model cannot receive the same training process as the smaller one. The relevant question is not whether the larger model is more capable in isolation. It is whether the improvement purchased by its extra parameters is worth the data exposure and training opportunities that must be given up elsewhere.
Even the phrase “model size” needs care. The first study focuses on non-embedding parameters because that definition produced cleaner relationships in its experiments. The Chinchilla analysis uses a different FLOP and parameter accounting that includes embedding contributions, although those contributions are relatively small for large models. This does not invalidate either study, but it means that a number copied from one frontier cannot be compared mechanically with a number from the other. A scaling claim is meaningful only together with its unit definitions, training objective, and budget convention.
The two supplied studies make this tension unusually visible. The earlier study by Kaplan and colleagues built an empirical framework for predicting how language-model loss changes with scale. Its compute-efficient recommendation put most additional budget into a larger model, with comparatively modest growth in data and serial training steps. The later Chinchilla study revisited the same fixed-compute problem with many more combinations of model size and training horizon. It concluded that the models then dominating attention were often too large for the amount of data used to train them, and that a smaller model trained on substantially more tokens could perform better at a similar training cost.
The important point is not that one paper offers a permanent replacement for the other. Both are attempts to estimate a frontier: the best performance that can be obtained when the total budget is fixed. Their different answers show that the frontier depends on what is measured, how the training runs are designed, and which regime the evidence actually covers. Understanding the disagreement is therefore more useful than memorizing either headline.
Three resources, three different outcomes
The basic resource map uses three quantities. Model size, usually represented by the number of non-embedding parameters N in the first study, describes the model's learned capacity. Dataset size, represented by training tokens D, describes how much text the model is exposed to. Training compute, represented by C, describes the floating-point work used to turn data into parameter updates. In the Kaplan analysis, a rough accounting is C ≈ 6NBS, where B is the batch size in tokens and S is the number of update steps. The exact accounting is a convention of the study and architecture, not a universal property of hardware.
These quantities are connected, but they are not interchangeable. Increasing N changes the learner. Increasing D changes the evidence from which it learns. Increasing C gives the training process more opportunity to use either resource, but it does not decide how that opportunity should be divided. A larger model trained on too few tokens may be data-limited; a huge dataset paired with too little capacity may be capacity-limited; and either configuration can be poorly optimized if the schedule or number of updates is unsuitable.
This also explains why a training budget should not be described as a single pile of interchangeable resources. Some compute buys representational capacity, some buys additional exposure to examples, and some buys more serial updates or parallel throughput. Two runs can report similar total FLOPs while making different compromises among these uses. The studies estimate the consequences of those compromises from observed loss curves; they do not assume in advance that one allocation is best.
The principal metric in both studies is token-level cross-entropy loss. In plain language, the model is rewarded for assigning high probability to the next token that appears in a sequence. Lower loss means better prediction on the measured text distribution. This metric is valuable because it is continuous and can be compared across many model sizes and training budgets. It is also narrower than the word “capability.” A model can improve its average prediction loss while changing unevenly on particular tasks, and smooth aggregate improvement can conceal qualitative changes—or failures—in behavior.
That distinction matters when the results leave the training curve. Downstream evaluations measure things such as question answering, reading comprehension, or common-sense benchmarks. They can show whether an allocation produces useful improvements, but they introduce their own prompt formats, task distributions, and contamination risks. Deployment adds another outcome: a smaller model may require less memory and less computation for every inference, even if its pre-training budget was the same as that of a larger alternative. The best allocation for minimizing training loss is not automatically the best allocation for serving millions of requests.
There is no contradiction in saying that loss is an important guide and an incomplete destination. It is useful precisely because it gives researchers a smooth signal with which to compare many runs. But a smooth signal cannot answer every question about a system. A model may improve on average while some tasks improve less, stay flat, or regress; the later evaluations in the Chinchilla study make that variation visible. The practical discipline is therefore to use loss to estimate the frontier, then test the outcomes that justify the expenditure.
The same discipline applies to cost. A large model may be attractive when the goal is to minimize serial training time and enough parallel hardware is available. A smaller model may be attractive when the goal is repeated inference or fine-tuning on constrained hardware. Neither choice follows from parameter count alone. The objective determines which part of the resource map deserves priority.
The diagram's central constraint is the fixed budget. Parameters and tokens are the main choices inside it; loss is the primary measured signal; task performance, inference cost, and data risks are consequences that must be checked separately. This is why “scale up” is incomplete advice. A useful scaling decision says what is being scaled, what is held fixed, and which outcome is being optimized.
In this setting, a compute frontier means the best observed or estimated loss available at a given budget after comparing alternative allocations. It is not a ranking of models by size, and it is not a fixed parameter-to-token ratio. Moving along the frontier changes both the model and the training process. The shape of that boundary is learned from experiments, so a frontier can move when the experiment covers a new range, uses a better schedule, or measures a different outcome.
Two frontiers, one continuing question
The first study found smooth, power-law relationships across broad ranges of model size, data size, and compute. A power law does not mean that every additional unit produces the same gain. Its small exponents describe diminishing returns: doubling a resource improves the loss by a modest amount, so continued progress becomes increasingly expensive. Within the tested Transformer regime, overall scale mattered more than many choices about depth, width, or attention-head shape. That finding made it reasonable to ask how a fixed budget should be divided rather than to treat every architectural detail as equally important.
The later study accepted the fixed-budget framing but changed the way the frontier was measured. It trained hundreds of models over different horizons, compared models at matched FLOP budgets, and calibrated the learning-rate schedule to the amount of training. Its estimates put model size and training tokens on approximately equal scaling paths as compute increased. A 70-billion-parameter Chinchilla model trained on 1.4 trillion tokens then outperformed the much larger 280-billion-parameter Gopher on many reported evaluations at a similar training compute budget. That case makes the allocation question concrete, but it does not make every other variable disappear: the models also differed in optimizer, tokenizer, and precision handling, and the evaluation evidence has limits.
Reading the studies together therefore requires two kinds of restraint. We should not dismiss the newer result because the experiments are imperfect; the matched-budget case is meaningful evidence that an under-trained large model can lose to a smaller, more data-trained one. But we should also not turn that result into a one-variable law saying that tokens alone explain every improvement. The useful conclusion is narrower and stronger: allocation is consequential, and the allocation must be tested under a clearly described training setup.
The question guiding this series is therefore conditional: how should a team allocate a particular compute budget between model capacity and training data, given its objective, training regime, data quality, and deployment constraints? To answer it, the next step is to define exactly what the scaling curves measure and why their apparently simple exponents can support such different decisions. Once those quantities are clear, the series can examine the bottlenecks, reconstruct the first frontier, understand why Chinchilla moved it, and return to the practical judgment that matters: not whether bigger is good, but when more model is worth more than more data.