When More Parameters Stop Helping
Once model size, training tokens, and compute are separated, the harder question is how they interact. This chapter examines the joint bottlenecks that appear when a Transformer grows larger than its data supply or when its training process uses a budget inefficiently. The goal is practical diagnosis: before adding parameters, a reader should be able to ask whether the active limit is capacity, data, or optimization—and whether “train longer” is buying useful progress or merely more cost.
The data bottleneck is a property of the pair
A model-size curve measured with abundant data answers one question: how much does additional capacity improve prediction when data is not limiting? A dataset-size curve measured with a sufficiently large model answers another: how much does additional text improve prediction when capacity is not limiting? Real training runs usually sit between those idealized conditions. The model and the dataset form a pair, and the pair can run into a limit in either direction.
Kaplan et al. describe that interaction with a joint loss relationship of the form
L(N,D) = [(Nc / N)^(αN / αD) + Dc / D]^αD.
The equation is easier to understand through its boundaries. If D becomes very large, the data term becomes small and the expression approaches the model-size relationship. The model then reveals how much the architecture can gain from more parameters when data is effectively abundant. If N becomes very large, the capacity term becomes small and the expression approaches the dataset-size relationship. In that limit, adding parameters cannot compensate for the information available in the finite dataset.
The equation is not saying that the model literally switches between two unrelated modes. Both contributions are present, but one can dominate. A model may be large enough that adding parameters produces only a small improvement because the data term is now the floor. Or it may have access to much more text than its capacity can use, so adding data produces little improvement until the model is enlarged. The useful question is which term is controlling the observed loss at the proposed scale.
This is the mechanism behind overfitting in the study's scaling picture. If a fixed dataset is paired with increasingly large models, the training process can continue to exploit capacity while held-out performance stops improving at the same rate. The larger model has not become useless in an absolute sense; it has become mismatched to the amount of data available for learning and evaluation. Conversely, a small model can remain capacity-limited even when the dataset is expanded, because it cannot represent enough of the patterns that additional text exposes.
Kaplan et al. find that the magnitude of the overfitting penalty is governed largely by a combination equivalent to N^0.74 / D. The practical implication is sublinear data growth: increasing model size does not require a proportional increase in tokens to keep the observed overfitting penalty controlled. Their estimate for staying within roughly the variation seen across random seeds is D ≳ (5 × 10^3)N^0.74. That relationship is useful as a way to reason about direction and relative growth, not as a universal data quota.
There are several reasons not to read that estimate too literally. The smallest datasets in the experiments fit poorly, and the study did not fully optimize regularization or data augmentation while varying model and dataset size. A token count also says nothing by itself about whether the text is redundant or useful for the intended task. The equation describes the reported loss behavior under the study's data and tokenization assumptions. It does not prove that every collection of the same number of tokens will support the same model.
The pairwise view changes how a scale-up proposal should be described. “We can afford twice as many parameters” is not enough. The proposal should also say whether the dataset grows, how many useful tokens are available, how the held-out distribution is defined, and which term in the joint relationship is expected to improve. If those questions are unanswered, the parameter increase may be purchasing capacity that the training setup cannot use.
Training time changes what compute can buy
Model size and dataset size still do not fully describe a run. A model can see a given amount of data through different numbers of parameter updates, batch sizes, and degrees of parallelism. These choices affect whether a fixed amount of computation is spent on more serial progress, more examples per update, or a faster route to a target loss.
Kaplan et al. fit the loss in the infinite-data limit with a model-size and effective-step relationship of the form
L(N,Smin) = (Nc / N)^αN + (Sc / Smin)^αS,
where Smin is an estimate of the minimum number of update steps needed to reach a given loss when batch size is very large. The fitted step exponent is about 0.76. The point of Smin is not that every useful run should use the largest possible batch. It provides a common way to compare training processes that reach the same loss with different combinations of steps and examples.
The critical batch size, written Bcrit(L), is the bridge between those combinations. It depends mainly on the target loss L, not directly on model size in the reported experiments. Below the critical batch size, increasing the batch can reduce the number of serial updates without greatly reducing compute efficiency. Above it, increasing the batch continues to reduce the number of steps but yields diminishing returns in total compute. The same loss can therefore be reached with different wall-clock and FLOP profiles.
This distinction is easy to lose in ordinary discussions of training speed. Suppose two teams reach the same target loss. One uses a smaller batch and more updates; the other uses a larger batch and fewer updates. If both operate near the useful range of the critical batch size, their total compute may be similar while their hardware requirements and elapsed time differ. A team with abundant parallel hardware may value fewer serial steps. A team constrained by total FLOPs may prefer not to push the batch beyond the point where it stops buying compute efficiency.
The study defines Cmin as an estimate of the minimum compute needed to reach a target loss when the batch is far below the critical size, and Smin as the corresponding minimum-step view when the batch is far above it. These are not three different kinds of model quality. They are ways of correcting for batch-size inefficiency so that model-size and training-time comparisons are less confounded by a particular batch choice.
The sequence is deliberately simple: each update consumes a batch, produces a loss signal, and changes the model. The critical batch size does not represent a new model component. It is a measured point in the trade-off between how much data is processed per update and how many updates must occur serially. Because that point moves with the target loss, it is not a single hardware setting that can be chosen once for every stage of training.
Why larger models can be more sample-efficient
The joint model/data relationship explains why a large model can be data-limited. The training-time relationship explains a different, complementary observation: once a target loss is specified, larger models can often reach it with fewer optimization steps and fewer processed examples than smaller models. This is the sample-efficiency result reported by Kaplan et al.
The intuition is not that larger models learn magically from no evidence. A larger model has more capacity to represent useful structure, so each update can move it toward a lower target loss more effectively within the measured regime. A small model may need many more updates to approximate the same predictive behavior, and it may still plateau at a higher loss. The larger model can therefore use a larger share of the compute budget on parameters and parallel data processing rather than on long serial training.
That advantage has a cost. More parameters require more memory and more computation per token. A larger model may reduce the number of steps but increase the amount of hardware needed to hold and update the weights. “Sample-efficient” means fewer examples or steps to reach a given loss; it does not mean cheaper in every resource, faster on every cluster, or cheaper to serve. The objective matters. Minimum training FLOPs, minimum wall-clock time, and minimum inference cost can favor different points on the model/data frontier.
This is also why convergence is not a neutral reference point. Conventional training practice often aims to continue until the loss appears close to its lowest attainable value for that model. Kaplan et al.'s compute-efficient analysis instead asks how to reach the best loss available for a fixed budget. Under that objective, a larger model can be trained for fewer serial steps and stopped while it remains somewhat above its fully converged loss. The saved training time and compute can be more valuable than the final marginal improvement that extra convergence would have delivered.
Compute-efficient early stopping should not be confused with careless under-training. Careless under-training means that a chosen model has not received the process needed to perform well, perhaps because the schedule was too short or the budget was exhausted by an oversized architecture. Compute-efficient stopping is a comparison among alternatives: it asks whether a larger model stopped earlier reaches a better loss per FLOP than a smaller model trained longer. The distinction is about the objective and the counterfactual, not about whether the loss curve has reached its lowest visible point.
The study's fit is strongest after an initial training transient and is less trustworthy very early in a run. Its critical-batch relationship was measured over a particular loss range, and the authors caution against projecting it far outside that range. These boundaries matter because a frontier derived from late training may mislead a decision about a very short run, different training dynamics, or a model far beyond the observed scale.
From isolated curves to a budget decision
The combined picture is now more constrained than the phrase “larger models learn better” suggests. Increasing parameters can lower loss when capacity is limiting, but can encounter a data bottleneck. Increasing tokens can lower loss when data is limiting, but can encounter a capacity bottleneck. Increasing batch size can shorten serial training, but can lose compute efficiency beyond a loss-dependent point. Training toward convergence can improve a particular model, but may be inferior to reallocating the budget to a different model size.
A disciplined diagnosis asks four questions. First, which loss or task outcome is the target? Second, is the proposed model limited mainly by capacity, data, or optimization? Third, is the priority total compute, elapsed training time, or deployment cost? Fourth, which parts of the observed scaling relationship are actually within the tested regime? The answers do not produce a universal ratio, but they rule out several bad comparisons.
Most importantly, the data requirement and the compute-efficient training requirement need not grow at the same rate. The first study's finite-data analysis says how data must grow to control overfitting as models become larger. Its compute-efficient frontier asks how to spend a budget while optimizing loss. Those are related questions, not identical ones. The next chapter combines the training curves and batch correction into the first explicit frontier, explaining why its answer favored much larger models and relatively modest growth in data—and why the same paper already contained reasons to treat that answer as conditional.