CMU 07-280 Lecture 16: Unifying Logistic and Linear Regression with Maximum Likelihood

🌏 中文版

Lecture 16, MLE and Probabilistic Modeling, took place on March 17, 2026. Earlier lectures used squared error and cross-entropy. This one explains why: assume how observations are generated by unknown parameters, then select the parameters that best explain observed data. No public lecture-by-lecture recording exists; this reading uses the lecture note, pre-reading, Recitation 9, and HW9.

Official material and scope

The primary sources are the Maximum Likelihood lecture note, full MLE pre-reading, Recitation 9 solution, and HW9. The pre-reading also covers categorical and multivariate Gaussian models plus conditional likelihood; this article focuses on how MLE reconstructs earlier regression objectives.

The inherited problem: a loss is not selected from an arbitrary menu

MLE asks which θ, under a stated model, would make the observations most likely:

θ̂MLE = argmaxθ p(D|θ)

This is not p(θ|D). Likelihood fixes the data and treats parameters as function inputs; it does not introduce a parameter prior. A posterior would additionally require Bayes' rule and a prior.

Complete conceptual path: i.i.d., logs, and negative log-likelihood

If observations are independent and identically distributed given θ,

p(D|θ) = Πi p(y(i)|θ)

Products of small probabilities underflow and are awkward to differentiate. Log is strictly increasing, so it preserves the maximizer while converting products to sums:

log p(D|θ) = Σi log p(y(i)|θ)

Optimization convention often minimizes, so negating produces negative log-likelihood. Each move has an assumption: factorization needs independence, logs need positive values and monotonicity, and max becomes min through the negative sign.

For prediction, use p(y(i)|x(i),θ). The course notes that machine learning commonly calls this likelihood or MLE, although conditional likelihood is the precise term.

Reproducible derivation one: MLE for a biased coin

Flip a coin n times, code heads as one, and let k=Σy(i). The Bernoulli likelihood is

L(θ) = θ^k (1-θ)^(n-k)
ℓ(θ) = k log θ + (n-k) log(1-θ)

Setting the derivative to zero gives

k/θ - (n-k)/(1-θ) = 0
θ̂ = k/n

Sixty heads in one hundred flips yields 0.6. The familiar sample proportion follows from the Bernoulli model and independence rather than intuition alone.

Reproducible derivation two: two regression losses from two observation models

For binary labels with

p(y=1|x,θ)=σ(θᵀx)

the conditional negative log-likelihood is binary cross-entropy. Lecture 9's logistic regression is therefore Bernoulli MLE.

For continuous output with

y(i)=θᵀx(i)+ε(i), ε(i)~N(0,σ²)

the part of Gaussian log-likelihood involving θ is -Σ(y(i)-θᵀx(i))²/(2σ²). Maximizing it is equivalent to minimizing squared error. The loss reflects a noise assumption; changing the observation distribution changes the objective.

Recitation and homework connection

Recitation 9 practices multivariate Gaussian log-likelihood and Lagrange multipliers, supplying the constrained-optimization tool needed when categorical probabilities must sum to one. HW9 connects MLE to N-gram sparsity and smoothing, forming the bridge to the next NLP lectures.

Anonymous learners can reproduce the public derivations. Online components, notebook environment, and staff feedback remain restricted.

Extension: MLE does not validate model assumptions

MLE finds the best parameters only within a chosen model family. If coin flips are dependent or regression-noise variance changes with x, i.i.d. or homoscedastic Gaussian assumptions fail. An exact optimum cannot repair a wrong data-generation model.

The next lecture continues MLE and enters natural-language processing. N-gram models factor sequence probabilities into conditional terms, while unseen events create zero likelihood and motivate smoothing.

What to do tonight

Use ten coin flips to tabulate Bernoulli likelihood and log-likelihood for θ=0.1,0.2,…,0.9, confirming that both peak near the sample mean. Then write the conditional log-likelihood for one logistic-regression example and match every term to cross-entropy. Finally, list whether your own dataset genuinely supports the i.i.d. assumption.

References

• CMU 07-280 Maximum Likelihood lecture note
• Maximum Likelihood Estimation pre-reading
• Recitation 9 solution: MLE and Lagrange Multipliers
• HW9 written component