Metrized Deep Learning: Finding a Ruler That Fits the Whole Network's Weights | MIT 6.7960 Lecture 23
Gradient descent silently assumes the weight space is Euclidean, and for neural networks it mostly is not. Bernstein's approach picks a norm for each layer (a shape-rescaled spectral norm for linear layers) and computes a norm for the whole network from two combination rules; normalize updates in it and the optimal learning rate stops drifting with width and depth. Push the same idea to duality maps and a linear layer's update becomes 'set every singular value of the gradient to 1', which is also what the nanoGPT training speed record of the time used.