AI training just got 2x faster — by predicting which requests will take longest

Researchers built a system that speeds up the slow part of training large language models by noticing that similar prompts produce similar response lengths. Instead of waiting for all requests to finish before moving forward, the system predicts which ones will be slow, schedules them differently, and uses speculative decoding to accelerate generation. This cuts the slowest requests down by 72-94% and doubles overall training throughput.

Training modern AI models involves running thousands of text generation tasks in lockstep — everyone waits for the slowest one to finish before the next batch starts. That's a brutal bottleneck. This paper shows you can break that bottleneck by doing what any sensible person would do: notice that requests are not equal, predict which ones will stall, and handle them differently. The implication is direct: faster training means more iterations, which means better models, cheaper per-model training, and faster iteration cycles for anyone building on top of large language models. This is the kind of infrastructure improvement that feels invisible but compounds — teams that can iterate faster on model training tend to stay ahead.