AI math models fail in predictable ways—a test catches them without retraining

Researchers found that large language models claiming to do step-by-step reasoning actually have two recurring failure modes: early layers get stuck on the current step and ignore history, while deeper layers gradually stop paying attention to the reasoning and just recycle the last few thoughts. A simple fix applied at test time—without retraining the model—recovers some of the lost reasoning performance on math, science, and coding tasks.

Until now, when these models failed on hard multi-step problems, it was opaque why. This paper shows the failure is structural and measurable: information doesn't flow properly through the model's layers during reasoning. That matters because it means you can diagnose a specific broken part—not just accept the failure as inherent to the architecture. The fix works across different models and tasks without retraining, which is practically significant. It suggests that a lot of the 'reasoning' performance we think is missing may actually be recoverable through targeted layer-level interventions rather than requiring entirely new training.