A new mathematical language for how neural networks normalize gradients during training

Researchers developed a mathematical framework to describe spectral normalization — the technique neural networks use to stabilize training by normalizing gradients across entire matrix blocks instead of individual numbers. This gives theorists a way to reason about when and why different normalization strategies work, and connects Muon (a popular normalization method) to classical optimal transport theory.

Deep learning optimization is still mostly empirical — people try things and see what works. This paper gives theorists a formal language to compare normalization strategies, which matters because the difference between unstable training and stable training is often just the right gradient scaling rule. The connection to optimal transport theory opens the possibility that techniques from mathematics (developed for completely different problems about moving probability distributions) might reveal why certain normalizations work better than others.