Logistic Regression: Derive the gradient from Bernoulli likelihood | Stanford CS109 Lecture 20

🌏 中文版

Video status: Related supplementary video included; the original lecture recording has not been verified. Source details

This is article 21 in Reading Stanford CS109, covering Summer 2026 Lecture 20: Logistic Regression (Jul 28), taught by Chris Gregg. It follows the official worksheet, answer key, LLM Learning Guide, and reader.

The lecture remains L2. The three-page worksheet has P1–P6 plus an optional MAP challenge. The four-page key omits only the P5 pset7 code solution and fully answers the challenge. The guide has six concepts across three pages. Current slides are unavailable and video is Canvas-gated.

P1: Poisson MLE warm-up

The counts [3,1,4,2,5] sum to 15:

LL(λ) = -nλ + (Σxᵢ)ln λ + constant
dLL/dλ = -n + Σxᵢ/λ = 0
λ̂ = Σxᵢ/n = 3

-Σln(xᵢ!) is constant in λ; it shifts the objective vertically without moving its argmax.

P2: Sigmoid, prediction, and decision boundary

For θ=[-1,2,0.5], prepend x₀=1. At (x₁,x₂)=(1,2):

z = θᵀx = -1 + 2(1) + 0.5(2) = 2
ŷ = σ(2) = 0.8808

The 0.5 threshold predicts class 1. Since σ is increasing and σ(0)=0.5, thresholding probability at 0.5 equals thresholding z at 0. The undecided set is -1+2x₁+0.5x₂=0, or x₂=2-4x₁: a decision-boundary line, and a hyperplane in higher dimensions. A zero weight removes a feature from the score; a large negative weight is strong evidence against Y=1.

P3: One gradient-ascent step

For ([1,1,0],1) and ([1,0,1],0) at θ=0, both predictions are 0.5:

LL = 2ln(0.5) ≈ -1.386
∇LL = [0.5,0.5,0] + [-0.5,0,-0.5]
     = [0,0.5,-0.5]

With η=1, the new θ=[0,0.5,-0.5]. Predictions become 0.6225 and 0.3775, so each observed label has probability 0.6225 and LL_new=2ln(0.6225)≈-0.948, a gain of about 0.438. Equal opposite residuals cancel the intercept gradient.

P4: Where the gradient comes from

The assumption is Y|X=x ~ Bern(ŷ), with ŷ=σ(θᵀx):

P(Y=y|x) = ŷʸ(1-ŷ)^(1-y)
LL = yln ŷ + (1-y)ln(1-ŷ)

∂LL/∂ŷ = (y-ŷ)/(ŷ(1-ŷ))
∂ŷ/∂θⱼ = ŷ(1-ŷ)xⱼ
∂LL/∂θⱼ = xⱼ(y-ŷ)

The sigmoid factors cancel. Across data, sum xⱼ⁽ⁱ⁾(y⁽ⁱ⁾-σ(θᵀx⁽ⁱ⁾)). The residual supplies direction and magnitude; xⱼ assigns feature j its credit or blame.

P5: Logistic Regression code

Pset7 specifies η=0.0001, 1,000 steps, and 100% test accuracy on data where y=x₁. The worksheet asks for batch pseudocode:

repeat training steps:
    reset gradient accumulator
    for each example:
        compute ŷ
        for each parameter:
            gradient[j] += x[j](y-ŷ)
    update every θ[j] after all examples

Reset once per outer iteration and update after the examples loop. The public key withholds the requested weight under pset policy, so this article does not invent a run result.

P6: Conceptual rapid fire

• Logs prevent product underflow, turn products into sums, and preserve argmax.
• Linear regression can leave [0,1] and assumes the wrong real-valued noise story; logistic regression models a conditional Bernoulli probability.
• If Y and Xᵢ are independent, expect θᵢ≈0, though finite samples need not produce exactly zero.
• An added feature that is always zero changes neither predictions nor training; its gradient is zero.
• To model x₁x₂, append it as a feature and learn its weight with the unchanged gradient.

Optional challenge: Gaussian prior and MAP

For P(Y=1|x)=σ(θx) with θ~N(0,1), density at 0 is e²≈7.389 times density at 2. Dropping the Bayes denominator and taking logs gives:

θ_MAP = argmaxθ [log f(θ) + Σᵢlog f(yᵢ|θ)]
gradient = -θ + Σᵢxᵢ[yᵢ-σ(θxᵢ)]

The extra -θ pulls the parameter toward zero: Gaussian-prior MAP produces L2 regularization or weight decay. It matters most with little data.

Guide unit: Assumption and training objective

A binary supervised datapoint has features x and y∈{0,1}. The model outputs a probability; thresholding produces a class:

P(Y=1|X=x) = σ(θᵀx)
LL(θ) = Σᵢ[yᵢlog ŷᵢ + (1-yᵢ)log(1-ŷᵢ)]

Machine learning calls the negative objective binary cross-entropy loss. The log-likelihood is concave in θ, so properly configured gradient ascent reaches the global optimum.

How to use the LLM Learning Guide

Work through its six concepts: classification setup, sigmoid, logistic assumption, log-likelihood, gradient derivation, and training loop. Attempt each Test me first. Ask the model to catch confusion between probability ŷ and a thresholded class, then debug pseudocode with a misplaced accumulator reset or update.

Material boundaries

• The worksheet has P1–P6 plus a challenge; the four-page key omits only P5 and answers the challenge.
• The three-page guide contains six concepts and no extra numbered unit.
• Current slides are unavailable and video is Canvas-gated; only public artifacts are used.

Course video sources

Verified the official Summer 2026 archive: the course is recorded and its syllabus says recordings are available through Canvas. The archived homepage's Videos link redirects to Stanford Canvas login and requires course access, so there is no public Summer 2026 player. The 2022 Stanford Online public recording(s) below cover the same topic but come from a different offering with different lecture numbering; they are supplementary material, not Summer 2026 lecture recordings.

Original videos: Stanford CS109 Probability for Computer Scientists I Logistic Regression I 2022 I Lecture 24

Official sources:

• CS109 Summer 2026 — Canvas lecture recordings (login required)
• CS109 Summer 2026 — archived official schedule
• CS109 Summer 2026 — archived syllabus and recording policy
• Stanford Online CS109 2022 public playlist (different offering, supplementary)

Checked: 2026-10-10.

Content check: verified against the video transcript (2026-10-10): Read the transcript of 2022 Lecture 24, "Logistic Regression" (about 75K characters): it covers how Naive Bayes relates to logistic regression, the sigmoid, the log-likelihood and gradient ascent, matching this lecture. The transcript has no decision boundary or Gaussian prior/MAP material, so those sections in this post are not video content.

Update Log

• 2026-10-10: Added explicit video status and checked recording sources and access notes.
• 2026-10-10: Rechecked video status. Switched to related supplementary video: embedded the 2022 Stanford Online public recording on the same topic; the Summer 2026 original is behind Canvas sign-in and could not be verified.
• 2026-10-10: Checked the video content against its transcript. The video topic matches; the decision-boundary and MAP sections are not video content, as the check note says.

References

• Stanford CS109 Summer 2026 schedule
• Lecture 20 page
• Lecture 20 worksheet
• Lecture 20 answer key
• Lecture 20 LLM Learning Guide
• Probability for Computer Science reader