AI language models just halved their memory demand by ditching full vocabulary prediction

Researchers built a language model that uses a tree structure instead of predicting from the entire vocabulary at once, cutting peak GPU memory in half while keeping the same accuracy. This matters because it makes training large language models cheaper and faster — you can now fit the same capability into smaller, cheaper hardware.

The prediction layer in current language models is bloated — it often takes up 20% of all parameters and dominates memory usage during training. This tree-based approach exploits the fact that tokens aren't random; they cluster hierarchically, so you can predict a token's ancestor in a vocabulary tree instead of picking from 50,000 options at once. The practical effect: you train faster and cheaper on the same hardware. The constraint has shifted from memory to where it should be — the depth of the neural network itself. For a field where training costs keep climbing, this is a measurable efficiency gain that appears stable across model sizes.