Scaling Rules for Optimization — Spectral View, Feature Learning, Hyperparameter Transfer | MIT 6.7960 Lecture 7
Optimization is not an isolated numerical problem: view SGD spectrally, the magnitude of weight updates determines feature learning; Maximal Update Parameterization transfers LR/init across width, and the critical batch size sets the marginal return of trading compute for convergence.