Maximum Likelihood and Logistic Regression: Why Least Squares Is a Reasonable Choice | Stanford CS229 Lecture 3
Assume the errors are zero-mean, IID Gaussian noise, take the log of the likelihood, and maximum likelihood turns out to be last lecture's least squares. Run the same recipe with 0/1 labels and a sigmoid and you get logistic regression, whose gradient is still "error times input". The lecture closes by comparing Newton's method with SGD: far fewer steps, but each step costs n·d² + d³.