The world is being quietly rearranged by people who write very long documents.


The title they went with On the Expressive Power of Contextual Relations in Transformers Noisy translates that to

Mathematicians prove transformers can learn any word relationship perfectly


Researchers showed mathematically that transformer models (the AI behind ChatGPT) can learn to represent any kind of relationship between words, no matter how complex. This closes a theoretical gap—we now have proof that transformers aren't fundamentally limited in what they can express, which validates their use in language tasks.
For years, AI researchers built transformers that worked remarkably well in practice, but nobody had proven they could actually express any relationship they needed to—this proof fills that gap and shows the architecture itself isn't a hidden bottleneck.

If you insist
Read the original →