Hazard of Overfitting: How Noise, Data Size, and Model Complexity Combine | NTU Hsuan-Tien Lin Machine Learning Foundations Lecture 13
Overfitting is the process in which switching to a more powerful model lowers E_in but raises E_out. This lecture runs experiments with two learners, one who only knows polynomials up to order 2 and one who knows them up to order 10, and arrives at four causes: too little data, too much stochastic noise, too much deterministic noise (a target that is too complex), and excessive model power. It also explains why data with no noise at all can still be overfit. For remedies it covers data cleaning, pruning, and virtual examples first; regularization and validation are left to the next two lectures.