🌏 中文版
Video status: Related supplementary video included; the original lecture recording has not been verified. Source details
This is article 20 in Reading Stanford CS109, covering Summer 2026 Lecture 19 (Jul 23). The schedule calls it MLE, while the lecture page uses Maximum Likelihood Estimation; Chris Gregg is the instructor. This guide follows the current worksheet, answer key, LLM Learning Guide, and course reader.
The lecture remains L2. Its three-page worksheet has P1–P6 plus an optional challenge. The two-page public key deliberately omits P5 and the challenge because they are pset6 problems. The three-page guide has six concepts; its last page only continues Concept 6 and the wrap-up. Current slides are unavailable and video is Canvas-gated, so this article stays within public artifacts.
P1: Connect entropy to parameter learning
For a uniform distribution over four values:
H(X) = -4(1/4)log₂(1/4) = 2 bitsAn even yes/no split leaves two equiprobable values under either answer. Expected remaining entropy is 1 bit, hence information gain is also 1 bit. The review connects information to estimation: data is valuable when it narrows the plausible models.
P2: MLE for a Geometric
X ~ Geo(p) counts deploy attempts through the first success. The data [2,1,4,1,2] sums to 10. The i.i.d. assumption gives:
L(p) = ∏ᵢ(1-p)^(xᵢ-1)p = p⁵(1-p)⁵
LL(p) = 5ln p + 5ln(1-p)
dLL/dp = n/p - (Σxᵢ-n)/(1-p) = 0
p̂ = n/Σxᵢ = 1/x̄Thus p̂=5/10=0.5. Every deploy ends in exactly one success, making this total successes divided by total attempts.
P3: MLE for an Exponential
Assume API request gaps follow X ~ Exp(λ). The gaps [0.8,1.4,0.2,0.6] hours sum to 3:
L(λ) = λⁿe^(-λΣxᵢ)
LL(λ) = nln λ - λΣxᵢ
λ̂ = n/Σxᵢ = 1/x̄ = 4/3 ≈ 1.33 requests/hourλ is a rate and the sample mean is an average gap, so they are reciprocals. Shorter mean gaps should imply a higher estimated arrival rate.
P4: Calculus misses a boundary maximum
Samples [1.2,3.7,2.1] come from Uni(0,β). The likelihood must include its support:
L(β) = β⁻³ if β ≥ maxᵢxᵢ = 3.7
= 0 otherwiseWithin the feasible region, the derivative of LL(β)=-3ln β is always negative. There is no interior critical point; likelihood decreases from the left boundary, so β̂=maxᵢxᵢ=3.7. It cannot overestimate the true endpoint, and continuous samples are almost surely below β, giving downward bias. “Differentiate and set to zero” finds only interior stationary points; parameter-dependent support requires checking boundaries.
P5: Rayleigh scale for a wind farm
The worksheet parameterizes the density as:
f(w|θ) = (w/θ)exp(-w²/(2θ)), w ≥ 0
LL(θ) = constant - Nln θ - Σᵢwᵢ²/(2θ)
θ̂ = Σᵢwᵢ²/(2N)The ten wind speeds give θ̂=18.915. P5 is labeled pset6: mle_wind, and the public key says only “Problem set problem.” This derivation follows the public prompt, not a hidden key solution.
P6: A point estimate versus a belief distribution
Six successes among nine users give p̂=6/9=2/3≈0.667. Updating the Laplace prior Beta(2,2) gives:
p | data ~ Beta(8,5)
posterior mean = 8/13 ≈ 0.615MLE is the point maximizing observed-data likelihood; a posterior is a full belief distribution over the parameter. With little data, the prior pulls its mean toward 1/2. At 600 successes out of 900, MLE is about 0.667 and the posterior mean is 602/904≈0.666; four pseudo-observations matter little.
Optional challenge: Negative Binomial
Each X counts experiments through r=5 successes. For 100 samples:
L(p) ∝ p^(nr)(1-p)^(Σxᵢ-nr)
p̂ = nr/ΣxᵢThe supplied observations sum to 1450, so p̂=500/1450=10/29≈0.345. Each observation contains five successes, making this total successes divided by total trials. The challenge is another pset6 problem omitted from the public key.
Guide unit: Parametric models and likelihood
Bernoulli has θ=p, Poisson and Exponential use θ=λ, and Normal has θ=(μ,σ²). Fixing θ selects one distribution from a model family. For i.i.d. data:
L(θ) = ∏ᵢf(xᵢ|θ)Independence justifies the product; identical distribution gives each factor the same model and parameter. L(θ) fixes data and varies θ; it is not automatically a normalized distribution over parameters.
Guide unit: Logs, argmax, and gradient ascent
Because log is strictly increasing, argmaxθ L(θ)=argmaxθ log L(θ). It turns products into sums, simplifies differentiation, and avoids numerical underflow. argmax returns a maximizing parameter, not the maximum function value.
When a derivative has no closed-form root:
initialize θ
repeat:
θ ← θ + η∇LL(θ)A large step size can overshoot or diverge; a tiny one converges slowly. A non-concave objective may lead to a local maximum. P4 is different: its maximum lies on a support boundary.
How to use the LLM Learning Guide
Follow its six concepts: parametric models, likelihood, log-likelihood and argmax, classic MLEs, boundaries and gradient ascent, then MLE versus Bayesian estimation. Attempt each Test me prompt first. Ask the model to identify the first wrong step and justify the i.i.d. product.
For the wrap-up, request one problem chaining model and θ identification, likelihood and log-likelihood, a complete MLE derivation, and a boundary or numerical-optimization case. Seeing the solution first practices reading, not modeling.
Material boundaries
- The worksheet has three pages with P1–P6 plus an optional challenge; the two-page key omits both pset6 solutions.
- The guide has six concepts across three pages; page three adds no seventh concept.
- Current slides are unavailable and video is Canvas-gated; this article does not reconstruct inaccessible content.
Course video sources
Verified the official Summer 2026 archive: the course is recorded and its syllabus says recordings are available through Canvas. The archived homepage's Videos link redirects to Stanford Canvas login and requires course access, so there is no public Summer 2026 player. The 2022 Stanford Online public recording(s) below cover the same topic but come from a different offering with different lecture numbering; they are supplementary material, not Summer 2026 lecture recordings.
Original videos: Stanford CS109 Probability for Computer Scientists I M.L.E. I 2022 I Lecture 21
Official sources:
- CS109 Summer 2026 — Canvas lecture recordings (login required)
- CS109 Summer 2026 — archived official schedule
- CS109 Summer 2026 — archived syllabus and recording policy
- Stanford Online CS109 2022 public playlist (different offering, supplementary)
Checked: 2026-10-10.
Content check: verified against the video transcript (2026-10-10): Read the transcript of 2022 Lecture 21, "M.L.E." (about 71K characters): it covers likelihood, log-likelihood, argmax and finding the MLE by differentiation (Bernoulli, Poisson, Gaussian and others), matching this lecture. The transcript has no geometric, Rayleigh, boundary-solution or negative-binomial material and no gradient ascent, so the matching worksheet sections in this post are not video content.
Update Log
- 2026-10-10: Added explicit video status and checked recording sources and access notes.
- 2026-10-10: Rechecked video status. Switched to related supplementary video: embedded the 2022 Stanford Online public recording on the same topic; the Summer 2026 original is behind Canvas sign-in and could not be verified.
- 2026-10-10: Checked the video content against its transcript. The video topic matches; geometric, Rayleigh, boundary solutions and gradient ascent are not video content, as the check note says.
References
Loading...