Exponential Families, Generalized Linear Models, and Softmax: How Many Times Do You Re-derive for a New Error Distribution? | Stanford CS229 Lecture 4
If the distribution of y can be written as b(y)·exp(ηᵀT(y) − a(η)), one derivative of a(η) gives the expectation and two give the variance. Set η = θᵀx, and linear regression, logistic regression, and softmax all share one update rule: θ ← θ + α(y − h(x))x. The second half of the lecture uses a four-class example to cover softmax, cross entropy, and label smoothing.