Researchers build tool to predict which coding tasks will stump AI agents — cutting benchmark testing time

A new framework predicts whether an AI agent will succeed or fail at individual coding tasks without running expensive evaluations. This means benchmark designers can calibrate task difficulty in advance, cutting the computational cost of testing AI coding agents by orders of magnitude.

Right now, testing how well an AI agent performs on coding benchmarks is computationally expensive — you have to actually run the agent through every task to see what works. This work extracts patterns from existing benchmark data to predict outcomes ahead of time, which means faster iteration on benchmark design and cheaper evaluation cycles. The practical effect is that more researchers and companies can afford to build and test their own coding agents, and benchmark designers can spot task-difficulty problems before investing in expensive runs.