AI inference chip makers claim they can run big language models 40% faster using lower-precision math

Researchers at NVIDIA built a new way to run transformer models (the neural networks behind ChatGPT-style AI) using less precise numbers during computation, which cuts memory use and speeds up inference. The technique trades a tiny bit of accuracy for major speedup on next-generation GPUs, and the code is open-source.

Running large language models is expensive because the math happens in high precision, which eats memory bandwidth on GPUs. If low-precision math actually works in production without degrading output quality, the cost per inference query drops significantly. This is a pure engineering win, not a fundamental breakthrough — but if the speedup holds on real workloads, it changes the unit economics of AI API services.