virajitgp

A Hitchhiker's Guide To Scaling AI. Chapter - 002

Like how Moore’s Law flipped the script in the ’60s, Kaplan and co. changed how we think about AI. Two years after OpenAI dropped their paper, DeepMind—and even Hoffman himself—came out with their own spin. Let’s dive in.

The Basic Idea Behind Scaling AI

Imagine you're building sandcastles. If you want a bigger, more detailed castle, you need three things: more sand (parameters), more time to shape it (compute), and more beach area to work with (data). AI scaling laws are like equations that tell us how these three things relate to each other.

The fascinating discovery researchers made is that these relationships follow predictable patterns—power laws, specifically. When we plot them on a logarithmic scale, they make straight lines. This isn't random; it reveals something fundamental about how artificial intelligence learns.

The Two Major Scaling Laws

Kaplan's Law (2020)

In 2020, Kaplan and his team at OpenAI discovered something remarkable. They found that a language model's performance (measured by how well it predicts the next word) improves in a predictable way as you increase:

  1. The number of parameters (N)
  2. The amount of training data (D)
  3. The compute resources used ©

The key insight was these relationships follow power laws. If you plot them on log-log scales, they form straight lines. What's fascinating is these power laws held across many orders of magnitude.

Kaplan's law suggested that if you have limited resources, you should build bigger models rather than using more data. Specifically, if your compute resources increase 10x, your model size should increase about 5.5x, while your dataset should only increase about 1.8x.

The formula looked something like:

Chinchilla's Law (2022)

Two years later, researchers at DeepMind found something that contradicted Kaplan. They discovered that many models were undertrained - they had too many parameters and not enough data!

The Chinchilla law said that for optimal performance, you should balance model size and data size much more evenly. For every doubling of parameters, you should also double your training data. The optimal ratio they found was about 20 tokens of training data per parameter.

This was a big shift! It means that if you have a fixed compute budget, you're better off with a smaller model trained on more data than a massive model trained on less data.

Why the Difference?

Why did Kaplan and Chinchilla reach different conclusions? It comes down to methodology:

  1. Definition of parameters: Kaplan excluded embedding layers when counting parameters; Chinchilla counted everything.
  2. Model sizes studied: Kaplan looked at smaller models (up to 1.5B parameters); Chinchilla examined larger ones (up to 16B).
  3. Hyperparameter tuning: Chinchilla was more thorough with optimization.
  4. Irreducible loss: Kaplan assumed the minimum possible loss was zero; Chinchilla didn't.

Recent research by Gwen (and others) has suggested that both laws are correct under different conditions. Smaller models may follow Kaplan's law, while larger ones follow Chinchilla's law. There might be "phase transitions" where scaling behaviors change.

A Physical Analogy

Think of training an AI model like filling a bucket with water. The bucket's capacity is your model size, the water is your data, and the speed of filling is your compute.

The Intuition Behind Scaling Laws

Here's what makes scaling laws so fascinating: they reveal something fundamental about how machine learning works. The power-law relationship suggests that learning follows principles similar to what we see in other complex systems in nature.

Recent Work: Going Beyond Simple Scaling

Gwen's work, along with other recent research, has started to identify limitations in these simple scaling laws:

Broken Neural Scaling Laws (BNSL)

Not all improvements follow smooth power laws. Sometimes, at certain scales, we see sudden jumps in capabilities. This has been called "broken neural scaling laws," where performance doesn't just gradually improve but shows discontinuous leaps.

For example, a model might suddenly become much better at arithmetic after reaching a certain size, or develop reasoning abilities that weren't present in smaller versions.

Data Quality Effects

One of the most important recent findings is that data quality dramatically affects scaling behavior. High-quality, diverse data allows models to achieve much better performance with the same parameter count.

Gwen's work has shown that filtering training data for quality can effectively "bend" the scaling curves, getting better performance than the original laws would predict.

Hitting the Limits: The Three Walls

As we push scaling laws further, researchers like Gwen have identified three major "walls" that may eventually limit our ability to keep scaling:

  1. The Data Wall: We're running out of high-quality human-created content. There's a finite amount of good data on the internet.

  2. The Compute/Energy Wall: Training increasingly massive models requires enormous computational resources and energy. The environmental and economic costs are substantial.

  3. The Architecture Wall: Current architectures like Transformers may have fundamental limitations that prevent them from achieving certain capabilities no matter how large they grow.

Scale-Time Equivalence

One fascinating recent discovery is what's called "scale-time equivalence." It suggests there's a trade-off between model size and training time. You can sometimes get similar performance by:

This means organizations with limited resources might still achieve competitive results by training smaller models more thoroughly.

What This All Means

These scaling laws aren't just academic curiosities—they're the reason why we have increasingly capable AI systems like Claude, GPT-4, and others.

Understanding scaling laws helps us:

  1. Predict progress: We can estimate how much better models will get with more resources
  2. Allocate resources efficiently: Organizations can make informed decisions about investment
  3. Identify bottlenecks: We know where the biggest improvements will come from

And perhaps most importantly, they tell us something profound about intelligence itself: that capabilities emerge in predictable ways as systems scale, suggesting patterns that may apply beyond just artificial neural networks.

The Future: Beyond Simple Scaling

The most exciting frontier now is finding ways to "bend" these scaling laws—to get better performance than the laws predict. This might involve:

  1. Better data curation: Being smarter about which data we train on
  2. Improved architectures: Designing models that scale more efficiently
  3. Novel training methods: Finding ways to extract more learning from the same data