What Scaling Laws Actually Measure
Scaling laws for autoregressive Transformer language models are empirical descriptions of how measured loss changes as model size, training tokens, and compute change. The specific question here is what those curves mean—and what they do not mean—when someone compares two training plans. By the end of this chapter, a reader should be able to interpret a fitted exponent, understand why the studies use different definitions of model size and compute, and recognize when a smooth relationship is evidence for a useful regularity rather than permission to extrapolate indefinitely.
Reading a scaling curve
The central metric is cross-entropy loss. An autoregressive language model reads a sequence of tokens and assigns a probability to the next token. If it assigns high probability to the token that actually appears, the penalty is small; if it assigns low probability, the penalty is large. The studies average this penalty over tokens and report it in nats. Lower loss means better next-token prediction on the distribution being measured.
That definition gives the research a useful measuring stick. Accuracy on a single task can jump in discrete steps, and a benchmark may be too small to distinguish nearby systems. Loss supplies a continuous signal that can be tracked across many training runs. It lets researchers ask whether a larger model improves smoothly, whether more data continues to help, and how much compute is needed to reach a particular level of prediction.
But loss is still a measurement of a particular behavior. It is not a direct reading of intelligence, reliability, or social value. A model can achieve lower average loss while improving unevenly across tasks. A loss curve can help estimate which allocation is efficient, but it cannot by itself certify that the resulting model will answer a particular question accurately or behave safely in deployment. The later evaluation of Chinchilla makes this distinction concrete: it reports broad gains, but not an improvement on every individual task.
The main variable for model size is N, the number of non-embedding parameters in the Kaplan study. A parameter is a learned numerical value adjusted during training. The parameter count is a rough measure of the capacity available to represent patterns, but it is not the same as the amount of knowledge stored in a simple database. The architecture determines how those values interact, and the training data and optimization process determine what they become.
The first study excludes vocabulary and positional embeddings from its primary model-size measure. That choice is not a claim that embeddings are irrelevant. It is a measurement decision: when the researchers compared models with different depths and shapes, the relationships were cleaner when they focused on non-embedding parameters. The Chinchilla analysis uses a different accounting convention, including embedding contributions in its total parameter and FLOP calculations. For large models those contributions are relatively small, but the difference still matters when readers compare numerical budgets across papers. A parameter count is meaningful only when its definition is attached.
The other two main coordinates are training tokens D and training compute C. Tokens are the units of text consumed by the model. Compute is the floating-point work used to process those tokens and update the parameters. Kaplan et al. use the approximation C ≈ 6NBS, where B is batch size in tokens and S is the number of parameter-update steps. The factor is a practical estimate for their Transformer training runs, not a universal conversion between parameters and hardware time. Chinchilla uses a more detailed all-FLOP accounting, so numerical comparisons across the studies require their definitions to be checked.
These definitions make the allocation problem precise. Increasing N changes the capacity of the learner. Increasing D gives the learner more text exposure. Increasing C gives the training process more work, but does not say how that work should be split. If a model is too small, additional tokens may have diminishing value. If it is too large for the available data, extra parameters may sit behind a data bottleneck. If the schedule is poorly matched to the intended horizon, a nominally large compute budget may not produce the best loss that the budget could have purchased.
A power law is a compact way to describe these diminishing returns. In the simplest model-size case, the loss behaves approximately like
L(N) = (Nc / N)^αN.
The notation is less important than the direction of the relationship. As N increases, loss decreases, but the exponent αN controls how quickly it decreases. A small exponent means that multiplying the resource produces a comparatively modest improvement in the fitted loss. The same form can describe loss as a function of dataset size or compute when the other resources are sufficiently large that they are not the active bottleneck.
Kaplan et al. report principal exponents of about 0.076 for model size, 0.095 for dataset size, and 0.050 for compute after their compute-efficient adjustment. These values are not scores on a common “better variable” scale. Each belongs to a different curve with its own conditions and fitted normalization. Their practical message is that all three axes matter and all three show diminishing returns. The cost of another improvement rises because the curve becomes flatter as scale increases.
An exponent is best understood as a sensitivity within a specified curve. If the model-size term is multiplied by a factor, the fitted contribution to loss changes by that factor raised to the exponent; with a small exponent, even a large resource increase produces a comparatively modest proportional change. This is why “the compute exponent is smaller” does not by itself prove that compute is less important than parameters or data. Compute is the budget that enables the other choices, and its curve summarizes an optimized allocation under particular conditions. Comparing the raw exponents as if they were prices on the same market would confuse separate experiments.
The equations also describe trends, not exact predictions for every run. Individual training results vary, and the fit is most informative when the run resembles the data used to estimate it. A curve can guide a decision between nearby plans while still being unreliable many orders of magnitude away. The farther a proposed model or dataset lies beyond the observed range, the more the conclusion depends on assumptions about architecture, optimization, data distribution, and the future shape of the curve. A fitted exponent is therefore a starting point for planning an experiment, not a substitute for one.
This is why a small loss change should not be dismissed as meaningless. A smooth metric can hide a large amount of engineering work: moving to a lower-loss region may require multiplying parameters, tokens, or FLOPs by a substantial factor. Conversely, a small average change can correspond to useful gains on some capabilities, especially when a model crosses a threshold in a particular task. The scaling curve tells us how expensive the average improvement appears to be; it does not tell us how every capability will move.
The fitted normalization constants also require restraint. Kaplan et al. note that quantities such as Nc and Dc depend on vocabulary and tokenization, so they do not have a fundamental meaning independent of the representation of text. Changing the tokenizer can change the number of tokens used to express the same material and can shift the numerical scale without changing the underlying language. Exponents and comparisons within a consistent setup are generally more informative than copying a normalization constant into a different setup.
The diagram represents a condition that is easy to miss. A one-factor curve is useful only when another resource is not already limiting performance. If the dataset is too small, increasing N will not reveal the infinite-data model-size curve. If the model is too small, adding tokens will not reveal the data-size curve. Scaling laws are therefore not independent dials whose effects can simply be added without checking the regime.
What the curves do—and do not—say about architecture
One striking result in Kaplan et al. is that, within a wide tested range, performance depended much more on total non-embedding scale than on the exact shape of the Transformer. The researchers varied depth, width, feed-forward dimension, and attention-head configuration while holding parameter count approximately fixed. The loss changed only a few percent across broad shape changes, although extremely shallow models and extreme depth-to-width ratios could depart from the common trend.
The correct interpretation is narrower than “architecture does not matter.” The experiments support a practical claim about the tested dense Transformer regime: once a model has a reasonable shape, total scale is often the stronger predictor of loss than the precise division between depth and width. That can simplify early planning, because a team need not assume that every architectural ratio deserves the same attention as the total budget. It does not remove the need to test architecture when the model leaves that regime, when hardware makes one shape faster, or when the objective changes.
The experimental comparison is informative because it holds the broad resource constant while changing the internal shape. If two models with similar non-embedding parameter counts have similar loss, the result weakens the idea that one particular depth-to-width choice is the main source of the observed scaling trend. It does not show that the shapes are interchangeable in every practical sense. The study does not test every practical training or serving consequence of those shape choices, nor behavior on tasks not represented by the loss measurement. A shape that is statistically similar on loss is not automatically interchangeable in every context.
The same caution applies to generalization across text distributions. Kaplan et al. trained models on one principal distribution and evaluated them on several other text distributions. Loss on those distributions improved smoothly with model size, broadly in parallel with the improvement on the training distribution. This is evidence that the training loss can serve as a useful signal for related text domains, rather than being meaningful only on the exact data used for fitting.
It is not evidence that every downstream task must improve by the same amount. A different text distribution is still a language-modeling evaluation, while question answering or reasoning benchmarks add task formats and behavioral demands. The studies support transfer of a trend within related text settings more directly than they support a universal law of capability. A lower loss should increase confidence that the model has improved in a broad predictive sense; it should not replace task-specific measurement.
The scope also matters for claims about other systems. The evidence concerns dense, autoregressive Transformers and primarily pre-training loss. Chinchilla reports a similar approximate relationship between model size and training tokens on C4 and GitHub, which strengthens the case that the allocation direction is not peculiar to one text mixture. But both papers leave open how the frontier changes for mixture-of-experts systems, retrieval-augmented models, multimodal systems, post-training, or a deployment objective dominated by inference cost. Those systems change what counts as active capacity, accessed data, or useful compute.
A scaling curve is therefore best read as a conditional statement with four parts: the metric, the variables, the regime, and the boundary. “Loss falls as model size rises” is incomplete. A more defensible statement says that, for the dense Transformer models and text distributions tested, when data and compute are not the bottleneck, loss follows a smooth decreasing relationship with non-embedding parameter count over the observed range. That sentence is longer, but its conditions are what make the claim useful.
The next difficulty follows directly from this conditional structure. The one-factor curves assume that the other resources are sufficiently available, yet real training runs have finite datasets and finite time. Once model size and token count grow together—or one grows much faster than the other—the bottleneck moves. The next chapter examines that joint relationship, including overfitting, sample efficiency, batch size, and early stopping. Those mechanisms explain why a scaling curve can look smooth in isolation while a fixed-budget decision still turns on which resource runs out first.