Scaling Laws

One of the most consequential empirical discoveries in AI research is also one of the simplest to state: when you make a model bigger, train it on more data, or spend more compute, it gets better, and the improvement follows a predictable mathematical relationship. This is a scaling law. Scaling laws transform AI development from a craft guided by intuition into something closer to an engineering discipline. If you can predict, before spending millions of dollars on a training run, approximately how well the model will perform, you can make rational resource allocation decisions. You can decide whether to invest in more data, a larger model, or a longer training run. You can compare the cost-effectiveness of different architectural choices. Scaling laws gave AI labs a map where previously they had only compass bearings.

The Kaplan et al. Findings (2020)

In 2020, a team at OpenAI led by Jared Kaplan published a landmark paper establishing scaling laws for neural language models. Their key finding: the test loss of a language model follows a power-law relationship with each of three independent variables: model size N (number of parameters), dataset size D (number of tokens), and compute budget C (floating-point operations used for training). A power law means that if you plot performance against scale on a log-log graph, you get a straight line. Concretely, each order-of-magnitude increase in model size produces a consistent, predictable improvement in loss, regardless of the starting scale. Double the scale again and you get the same amount of improvement. The Kaplan team found that model size was the dominant factor: given a fixed compute budget, you should use most of it to make the model larger while training it on relatively less data. This finding directly shaped GPT-3: a very large model of 175 billion parameters trained on a relatively modest 300 billion tokens. Equally important was what did not matter: architecture choices within the transformer family, such as the ratio of layers to width or number of attention heads, had surprisingly little effect on the scaling law. Scale, not architecture tweaks, was the primary lever.

In 2022, DeepMind researchers published a paper colloquially called the Chinchilla paper that refined the Kaplan findings with a critical correction: for a fixed compute budget, the recommendation to maximize model size while undertraining on data was not optimal. The Chinchilla team showed that model size and training tokens should scale together roughly equally. For every doubling of model parameters, you should approximately double the training tokens. The Chinchilla key result: a 70-billion parameter model trained on 1.4 trillion tokens outperformed a 280-billion parameter model trained on only 300 billion tokens, despite using four times fewer parameters. The larger model was undertrained on data. This finding reshaped the field. Most large models circa 2021-2022 including GPT-3 were overparameterized relative to their training data. They could achieve lower loss by training smaller models on much more data. The practical implication for deployment: the optimal model for real-world use is smaller than maximally compute-efficient because inference cost, the cost of running the model after training, also scales with model size. Post-Chinchilla, the industry converged on a different trade-off: train models smaller than maximally compute-efficient but on far more data than Chinchilla recommends, producing models that are cheaper to run at deployment while still highly capable. Llama 3 and Mistral models exemplify this philosophy.

What Scaling Laws Do Not Tell You

Scaling laws are powerful, but they have important limits that practitioners must understand. Scaling laws predict average loss on held-out text, a smooth statistical measure. They do not predict the emergence of specific abilities. A model might smoothly improve its next-token prediction loss as it scales, yet suddenly acquire the ability to perform multi-step arithmetic or write coherent code at a specific scale threshold. These jumps, called emergent capabilities, are not predicted by the smooth loss curve and are the subject of Lesson 5. Scaling laws are empirical, not theoretical. They are best-fit curves to observed data, not derivations from first principles. There is no proof that power-law scaling will continue across all scales or all domains. Historical data covers roughly four to five orders of magnitude of compute. Whether scaling continues to produce improvements at the next four orders of magnitude is an open empirical question. Scaling laws assume the data distribution stays consistent. The laws break down when the training data runs out of quality content, when the architecture changes fundamentally, or when publicly available high-quality text has been effectively exhausted. There is an active debate about whether synthetic data generated by models themselves can continue to fuel scaling. Finally, scaling laws say nothing about alignment or safety. A larger model is not necessarily safer, more honest, or more reliably aligned with human values. Performance on downstream tasks can improve dramatically with scale while alignment-related properties such as calibration, honesty, and robustness may improve only modestly or inconsistently.