Gaussian Discriminant Analysis and Naive Bayes: Model Each Class First, Then Classify with Bayes' Rule | Stanford CS229 Lecture 5
Lecture 5 turns classification around: instead of learning p(y|x) directly, it learns p(x|y) and p(y). GDA assumes the two classes are Gaussians with a shared covariance. Maximum likelihood then takes nothing more than counting proportions and averaging, the decision boundary comes out as a straight line, and the posterior is exactly a sigmoid. Naive Bayes uses conditional independence to squeeze 2^d parameters down to 2d+1.