Generalization Theory: With Enough Capacity to Memorize the Data, Why Deep Nets Still Don't Just Memorize | MIT 6.7960 Lecture 6
Neither parameter count nor VC dimension explains generalization in deep nets: a network can memorize random labels to 100%, which makes the VC bound vacuous, and past the interpolation threshold test error falls a second time as parameters grow. Isola's account is that there are pressures beyond fitting the data, and he considers architectural symmetry and compositionality the most important. The question is still open.