Language models could predict text as continuous math instead of picking one word at a time

Researchers propose a different way for AI language models to generate text: instead of selecting discrete words one by one, the model predicts smooth mathematical vectors in embedding space and only converts them to actual words at the end. This could give models more flexibility to refine their predictions before committing to words, and opens a wider design space for how text generation actually works.

The standard way all large language models work right now is locked into a single architectural pattern: pick the next token, commit to it, move forward. This paper shows that's not the only way to organize autoregressive generation, which matters because if the continuous approach produces better text or offers real computational advantages, it could reshape how the next generation of models get built. The paper also demonstrates that you can add control surfaces into the generation process itself—changing direction, adding noise, delaying commitment—before words become final, which opens room for steering and refinement that token-by-token selection doesn't easily permit.