A Hitchhiker's Guide To Scaling AI. Chapter - 001

09 Mar, 2025

Kaplan et al. (2020)

Historical Context: From Moore's Law to AI Scaling Laws

In 1965, Gordon Moore observed that "the number of transistors that could be placed on an integrated circuit (IC) was doubling roughly every 2 years." This simple empirical observation became the foundation for Intel's growth strategy and helped drive the entire computer revolution. Moore's Law wasn't just an interesting observation—it was a road-map that guided decades of semiconductor development.

Fast forward to 2020, and we see a similar paradigm shift in artificial intelligence. OpenAI's paper "Scaling Laws for Neural Language Models," with core contributors Jared Kaplan and Sam McCandlish, systematically investigated what many AI researchers had intuitively suspected: that model performance improves in predictable ways as we scale up resources, AKA Bigger is Better.

The Fundamental Scaling Variables (The Big 3)

Before Understanding the mathematical relationships, let's clarify the three fundamental variables in scaling laws:

1. Model Parameters [N]:

   • These are the trainable weights in the neural network
   • More parameters allow the model to capture more complex patterns
   • Measured simply as a count (e.g., 175 billion parameters in GPT-3)

2. Dataset Size [D]:

   • The amount of text used to train the model
   • Usually measured in tokens (word fragments)
   • Larger datasets expose the model to more patterns and knowledge

3. Compute [C]:

   • The computational resources used for training
   • Often measured in petaflop/s-days (a unit of computation)
   • Combines both the speed of calculation and duration of training

The Mathematical Relationships of Scaling Laws

The key insight from Kaplan et al. is that model performance follows power-law relationships with each of these variables. Let's break down each relationship in detail:

1. Scaling with Model Size [N]

$L(N)\propto N^{-{\alpha }_N}$

Where:

• $L$ is the cross-entropy loss (lower means better performance)
• $N$ is the number of model parameters
• ${\alpha }_N\approx 0.076$ is the power-law exponent
• $\propto$ means "proportional to"

This equation tells us that as we increase the number of parameters, the loss decreases following a power law. The exponent ${\alpha }_N=0.076$ means that doubling the model size reduces the loss by a factor of $2^{0.076}\approx 1.054$, or about 5.4% improvement.

Importantly, this relationship holds across many orders of magnitude—from models with millions of parameters to those with billions—suggesting a fundamental property of neural language models.

2. Scaling with Dataset Size [D]

$L(D)\propto D^{-{\alpha }_D}$

Where:

• $D$ is the dataset size (typically measured in tokens)
• ${\alpha }_D\approx 0.095$ is the power-law exponent

This equation shows that as we increase the amount of training data, the loss also decreases following a power law. The exponent ${\alpha }_D=0.095$ means that doubling the dataset size reduces the loss by a factor of $2^{0.095}\approx 1.068$, or about a 6.8% improvement.

3. Scaling with Compute [C]

$L(C)\propto C^{-{\alpha }_C}$

Where:

• $C$ is the compute used for training
• ${\alpha }_C\approx 0.050$ is the power-law exponent

This equation describes how loss decreases as we increase the computational resources devoted to training. The exponent ${\alpha }_C=0.050$ means that doubling the compute reduces the loss by a factor of $2^{0.050}\approx 1.035$, or about a 3.5% improvement.

This relationship assumes that compute is allocated optimally between model size and dataset size.

4. Optimal Allocation of Compute

One of the most practical insights from Kaplan et al. is how to optimally allocate compute between model size and dataset size:

$D\propto C^{0.27}$

This equation tells us that as we increase our compute budget, we should allocate roughly 27% to increasing the dataset size and the remaining 73% to increasing the model size.

The full derivation comes from:

$D\propto N\propto C^{\frac{1}{1+{\alpha }_D/{\alpha }_N}}$

Substituting the values ${\alpha }_D=0.095$ and ${\alpha }_N=0.076$:

$C^{\frac{1}{1+0.095/0.076}}\approx C^{\frac{1}{1+1.25}}\approx C^{\frac{1}{2.25}}\approx C^{0.444}$

Then, because $D\propto N^{{\alpha }_N/{\alpha }_D}\approx N^{0.8}$ and $N\propto C^{0.444}$, we get:

$D\propto (C^{0.444}{)}^{0.8}\approx C^{0.356}$

Kaplan et al. empirically found it closer to $C^{0.27}$, which is the value commonly cited.

5. The Universality of Overfitting

The paper also introduced the concept of the "universality of overfitting," described by:

$L(N,D)\propto \frac{N^{-{\alpha }_N}}{1-(N/D{)}^{\beta }}$

Where:

• $\beta$ is a constant related to the task

This equation elegantly captures two regimes:

1. When $N\ll D$ (model size much smaller than dataset size), the denominator is approximately 1, and we get the standard power-law scaling: $L(N,D)\propto N^{-{\alpha }_N}$
2. When $N$ approaches or exceeds $D$, the denominator decreases, causing the loss to increase—this is overfitting

The term "universality" refers to the finding that this pattern of overfitting appears consistently across model architectures and sizes.

Practical Implications of Scaling Laws

These mathematical relationships have profound practical implications:

1. Predictable Improvements: Organizations can reliably predict how much better their models will get with more resources.

2. Resource Allocation: The optimal allocation formula ($D\propto C^{0.27}$) provides clear guidance on how to divide resources between collecting more data and building larger models.

3. Diminishing Returns: The power-law relationships mean each doubling of resources yields a smaller percentage improvement than the previous doubling—informing cost-benefit decisions.

4. Architecture Agnostic: The finding that performance depends weakly on model architecture ("shape") but strongly on scale suggests that simple architectures scaled appropriately can outperform complex architectures with fewer resources.

The Empirical Evidence

What makes the Kaplan et al. paper so compelling is the extensive empirical evidence. They trained over 60 different Transformer language models, varying in size from 768 to 1.5 billion parameters, with between 22 million and 17 billion tokens of training data.

The consistency of the scaling trends across this wide range of conditions is what gives the scaling laws their predictive power. There were "NO SIGNS OF DEVIATION from these trends on the upper end," suggesting that these relationships would continue to hold for even larger models—a prediction that has been borne out by the subsequent success of models like GPT-3 and GPT-4.

Critical Insight: L(N, D) ∝ N^(-0.78)/D

The Landmark Insights in the paper are as follows:

$L(N,D)\propto \frac{N^{-0.78}}{D}$

This is a simplified form of the overfitting equation that appears when N approaches D. The exponent -0.78 combines the effects of both the power-law scaling with model size and the overfitting term.

This relationship tells us that:

1. Increasing model size (N) decreases loss with an exponent of approximately -0.78 (which is steeper than the -0.076 when N is much smaller than D)
2. Increasing dataset size (D) decreases loss linearly (with exponent -1)
3. The ratio between model size and dataset size is critical: if you increase model size without a proportional increase in dataset size, you'll hit diminishing returns due to overfitting

Scaling Laws Mathematical Relationships: Summary Table

Key Mathematical Relationships from Kaplan et al. (2020)

Derived Relationships

Units and Measurements

Bibliography:

• Scaling Laws for Neural Language Models