Training giant AI models on longer texts requires 150 billion tokens — far more than researchers thought

Researchers tested how long it takes to train industrial-scale language models to handle very long text inputs, and found that previous estimates were dramatically too optimistic. Companies training these models now know they need to budget roughly 150 billion tokens of data before the training actually converges, not the tens of billions that earlier studies suggested.

This is a cost and timeline signal disguised as a technical paper. If you're training a 13-billion-parameter language model on long-context tasks, you just learned your training budget needs to be 3 to 5 times larger than you planned. That means longer training runs, higher compute costs, and delayed product timelines for any company betting on next-generation long-context AI. The paper also exposes a measurement trap: the standard benchmarks everyone uses (like Needle-in-a-Haystack) report false confidence — they show the model is 'done' learning when it actually isn't. This means every company that relied on those benchmarks to decide when to stop training probably stopped too early.