The First Frontier: Why Bigger Models Looked Compute-Optimal

The earlier scaling study did more than report that larger language models usually have lower loss. It combined model-size, training-time, and batch-size relationships to ask a constrained question: if the total training compute is fixed, which model and training process should produce the best loss? This chapter reconstructs that first compute-efficient frontier, explains why it directed most new compute toward model parameters, and separates the empirical fit from the assumptions and extrapolations that made the recommendation conditional.

Turning learning curves into an allocation

The ingredients are the relationships established earlier. In the infinite-data view, loss improves as model size N increases. In the training-time view, loss also improves with the effective minimum number of steps Smin, with a fitted relationship of the form

L(N,Smin) = (Nc / N)^αN + (Sc / Smin)^αS.

This equation says that two sources of error remain: the model may not have enough capacity, or the training process may not have taken enough useful steps. The first term falls as N grows. The second falls as Smin grows. To compare models under a fixed budget, the researchers need to connect those terms to the compute spent on a run.

The connection runs through the critical batch size. At a target loss, training with a batch near Bcrit(L) provides a compromise between processing more examples per update and preserving compute efficiency. A rough compute accounting is then proportional to N × Bcrit × Smin, with the constant of proportionality determined by the study's FLOP convention. If the model becomes larger, each update costs more. If the target loss becomes lower, the critical batch size changes. If the training process uses more effective steps, it processes more examples and consumes more compute.

The fixed-budget problem can now be stated without treating “bigger” as a complete strategy. Choose a model size and a training process so that the compute constraint is met, then minimize the fitted loss. Increasing N lowers the capacity term but leaves fewer resources for training steps or data unless the batch and schedule are adjusted. Decreasing N permits more updates or examples, but eventually gives up too much representational capacity. The frontier is the boundary formed by the best trade-offs among those alternatives.

The result is not obtained by deciding in advance that parameters matter more than data. It emerges from the fitted shapes of the loss curves. Because larger models are more sample-efficient in the measured regime, the model-size term can fall rapidly enough that the best use of additional compute is to enlarge the model, then use batch size and modest changes in serial steps to train it effectively.

The optimization has a useful counterfactual hidden inside it. Imagine holding compute fixed while moving a small amount of budget from parameters to additional steps. The smaller model receives more opportunities to update, but its capacity term rises. Move the same amount in the other direction and the capacity term falls, but the larger model may receive too few useful updates.