Transformer AI models might be doing geometry, not just statistics

The core parts of a Transformer AI model, like attention and residual connections, can be explained by a single geometric math problem. This means these AI models might be estimating directions on a sphere, rather than just finding patterns in data.

For years, AI researchers have designed Transformer models by adding components like attention and normalization as separate choices. This paper suggests these components are not arbitrary design decisions. Instead, they naturally arise from the geometry of estimating a state on a curved surface. This could lead to new ways of building AI models that are more efficient or easier to understand.