Former OpenAI researcher Diogo Almeida published a post revealing that the original Scaling Law contained a bug, misleading the AI industry for several years.

Author and source: AI World

OpenAI has misled the entire AI community for years!

Over the past five years, the entire AI industry has been driven forward by the scaling law.

Ultraman's confidence in AGI stems from this curve.

Now, someone has come forward to say: This curve was wrong from the start.

Not a hindsight comment—this was said by Diogo Almeida, a researcher at OpenAI at the time who worked on large model optimization.

Just now, he published a blog post with a chilling title—“Scaling Laws, Honestly.”

The original version of the scaling law was wrong due to a bug.

Portal: https://www.completeskeptic.com/p/scaling-laws-honestly

DeepMind’s Sander Dieleman, who achieved fame with diffusion models, then tweeted about it, calling it an interesting anecdote about LLMs:

The original scaling law was incorrect due to a bug, likely causing the industry to waste massive computational resources on a multitude of models that were too large but undertrained.

A bug that burned through two years.

When the bug was exposed, what we saw was not just a black hole of computing power, but an intelligent boundary far deeper than imagined, reshaped by language itself.

The Scaling Law is the LLM version of the "geocentric theory"

In 2020, OpenAI concluded that, under a fixed compute budget, you should prioritize scaling up the model size rather than gathering more data.

In formula terms, the optimal number of parameters is proportional to the 0.73rd power of compute—parameters are the variable you should prioritize scaling.

This sentence directly defined the appearance of the GPT-3 generation: stack parameters, pile them up relentlessly—175 billion.

It tells developers around the world: Don’t ask questions—just stack more parameters; if you make the model large enough, miracles will happen.

Two years later, DeepMind released Chinchilla, completely overturning this conclusion: models and data should be scaled up roughly equally, with about 20 tokens per parameter being optimal.

They trained a Chinchilla model with 70 billion parameters using 1.4 trillion tokens—less than half the size of GPT-3 but more than four times the data.

As a result, with the same computational budget, it completely surpassed Gopher, which has 280 billion parameters but was trained on only 300 billion tokens.

The same amount of money—one turned it into a bloated, bulky guy, while the other trained it into a lean, fit boxer.

After a three-year delay, Peking University alumnus Weng Li provides an in-depth analysis of the mainstream explanations for the differences in subsequent research, namely that the discrepancy lies in how they calculate the total number of parameters.

And that’s not all—even the “correct” Chinchilla isn’t clean itself.

In 2024, Besiroglu et al. extracted the original data points from Chinchilla and re-ran them, discovering a bug hidden within their own fitting method:

The loss scale in the optimizer was set too high, and the Huber loss was averaged over samples instead of summed, causing premature termination of fitting.

The paper fixes one bug but introduces another.

At this point, the much-repeated phrase “first principles” suddenly seems a bit shaky.

The so-called Scaling Law is never a rigid physical law like Newton's three laws; it is merely an empirically fitted curve.

When Diogo Almeida believed the truth was otherwise—not that the approach was different, but that the original version of the scaling law had a bug.

Did OpenAI trick global AI peers with three tricks?

To create a lie that convinces the global AI community, you only need three steps.

Step 1: Lock the data.

The OpenAI paper fed the same amount of data—approximately 130 billion tokens—to all models, whether they were still learning to walk (small models) or had grown into giants.

Small models are thus overfed, even overwhelmed, while large models, which require vast amounts of data to fill their capacity, suffer severe undernourishment under the same token budget.

The Chinchilla paper later pointed out succinctly: they used a fixed number of training tokens and learning rate schedule for all models.

It’s like giving kindergarteners and PhD candidates the same exam at the same time, then claiming that “scores depend only on talent.”

Step 2: The self-deceptive LR decay.

They used cosine learning rate decay to smoothly reduce the learning rate to zero as training approaches completion.

As training approaches the preset endpoint, the learning rate is gradually reduced to zero, causing the model's progress to naturally plateau.

When the curve flattens, it looks like: the model has learned everything it can, and feeding it more data won't help.

The researchers therefore concluded: "Adding more data is useless; the model has already saturated."

This is not the model's limit—it's the learning rate artificially cutting short the model's growth. It creates a false impression that performance has hit a ceiling and more data won't help.

But now we know that those large models have not reached their limit.

Step three: The arrogance of authority.

Step three, and the most sneaky one: the paper stated that the results were "largely independent of the learning rate schedule."

Although many, including Diogo Almeida at OpenAI at the time, had a vague sense that something was off, the conclusion was technically correct under a fixed token limit.

But it偏偏 does not apply to the ideal world of "infinite data" that the scaling law truly aims to describe.

They mistook local truths under limited conditions for universal cosmic laws.

Stack three steps together, and you get a law that is both wrong and extremely difficult to debug.

Even Diogo himself admits: back then, he was also optimizing at OpenAI and didn’t spot this bug—the learning rate curve looked so obviously “carefully designed” that no one would have suspected anything.

GPU resources are being wasted, with severe misallocation of computing power.

Guided by OpenAI's flawed formula, the AI industry has entered the era of "brute force yields miracles."

This means that over the past few years, the world’s brightest minds and most scarce computing power have been wasted on ineffective scaling efforts.

This isn't just about money—it's about humanity collectively sprinting thousands of kilometers down the wrong track in the race toward AGI, all due to an incorrect learning rate setting.

If the discovery of the bug was heartbreaking, the deep reflection it triggered was chilling.

Researcher Adam Zachary Wasserman pointed out a blind spot overlooked by everyone: even with formula corrections, the current Scaling Law is merely an "English Scaling Law."

He conducted a counterintuitive experiment: training models with the same architecture and the same computational power.

It was found that the French model achieves a certain level of grammatical proficiency 50 to 100 times more efficiently than the English model.

Why? Because English is a "morphologically poor" language.

It relies too heavily on distributional patterns, requiring the model to guess word meanings from vast amounts of data; whereas languages like French or Chinese, with their rich morphology or structured syntax, carry abundant explicit information within the words themselves.

This means that all of our current hashing power allocation schemes are based on the most data-intensive and least efficient language.

When you think you're exploring the physical laws of "general intelligence," you're really just measuring how computationally wasteful the English language is.

It’s like trying to establish nutritional standards for all life in the universe by studying a pig’s appetite—not only is it biased, but it also reveals a limitation in your thinking.

We could have achieved better performance with a smaller model and more high-quality data.

We could have saved tens of thousands of hours of H100 runtime in power and heat.

We could have entered the era of "efficient AI" two years earlier.