AI researchers cut LLM inference speed in half by routing layers dynamically instead of treating all attention equally

A new attention method lets large language models skip expensive computation on some layers while keeping it on others, based on what the input actually needs. In practice, this means running the same model 2.8 times faster on long documents without retraining or losing accuracy.

The main computational bottleneck in LLMs is the attention mechanism, which has to compare every token to every other token—a cost that balloons with context length. This paper shows that you don't need full attention everywhere; some layers can use cheaper sparse approximations without hurting output quality. That matters because it translates a theoretical speedup into a measurable wall-clock improvement on actual hardware, which is where most attention research fails. The constraint is tight: the method only needs 12 hours of training on 8 GPUs, so it's practical to add to existing models.