Linear Regression and the Normal Equation | CMU 07-280 Lecture 7

🌏 中文版

Video status: Checked: no corresponding recording link listed on the public official page. Source details

This is CMU 07-280 Spring 2026 Lecture 7: Linear Regression. Lecture 5's ERM is solved end to end: choose linear hypotheses, choose squared loss, stack the data into a matrix, and derive a closed-form optimum.

The inherited question: what ERM becomes for a linear hypothesis class

Start with X=R, Y=R, and let the hypothesis class contain all lines:

h_(ω,b)(x) = xω+b

With squared loss, ERM finds ω,b minimizing average squared residual. To isolate the core argument, the notes first set b=0 and fit a line through the origin, then generalize to multiple dimensions and an intercept.

Full conceptual path: from a scalar to the design matrix

In the one-dimensional, through-origin case:

J(ω) = (1/N) Σ (y(i)-x(i)ω)²

Expanding or differentiating yields:

ω* = Σ x(i)y(i) / Σ x(i)²

In multiple dimensions, absorb the intercept into a constant-one feature and write hθ(x)=xᵀθ. Stack each x(i)ᵀ into a design matrix X and labels into y:

J(θ) = ||y-Xθ||²
     = yᵀy - 2θᵀXᵀy + θᵀXᵀXθ

Setting the gradient to zero gives the normal equations:

XᵀXθ = Xᵀy

If XᵀX is invertible:

θ* = (XᵀX)^-1 Xᵀy

The notes explicitly warn against writing θ=X^-1y: X is usually not square and may not be invertible. Even though XᵀX is square, linearly dependent features make it singular. The closed form carries an invertibility assumption.

A reproducible derivation: fitting two points through the origin

Use (x,y)={(1,2),(2,3)}:

Σxy = 1·2 + 2·3 = 8
Σx² = 1² + 2² = 5
ω* = 8/5 = 1.6

The model h(x)=1.6x predicts 1.6 and 3.2. Squared errors are 0.4² and (-0.2)², for a mean of 0.1.

Verify by differentiation:

J(ω)=[(2-ω)²+(3-2ω)²]/2
dJ/dω=5ω-8

Setting the derivative to zero again gives ω=8/5. This is the same minimum that gradient descent will approach iteratively in Lecture 8. Linear regression is special here because it can also be solved directly.

Recitation and homework connection

Recitation 4 builds scalar, vector, and matrix derivatives, then derives the gradient and closed form from J(θ)=||Xθ-y||². Its solution keeps every matrix shape aligned, a more durable skill than memorizing the normal equation.

HW4 asks students to expand a concrete dataset's objective into a polynomial, write partial derivatives, and design datasets with specified numbers of optimal solutions. Later parts derive weighted least squares and multi-output regression. The task checks whether the formula can be rebuilt from an objective rather than applied in one fixed form.

Further comparison: closed forms and numerical solutions have different roles

The normal equation gives an analytic answer and makes the geometry visible. Forming and inverting a large XᵀX, however, can be expensive and numerically delicate. Lecture 8 turns to gradient optimization not because linear regression became unsolvable, but because the course needs a method that extends to models without closed forms.

Both routes optimize the same J(θ). A closed form solves one equation; gradient descent repeatedly follows local slope.

What to do tonight

1. Compute Σxy/Σx² for three one-dimensional points and verify it by differentiation.
2. Write a design matrix with an intercept column and label the shapes of X, θ, and y.
3. Construct a matrix with two identical feature columns and explain why XᵀX is singular.

Course video sources

The official Spring 2026 schedule has been checked: it lists slides, notes and exercises, but no public recording link for the lectures covered here. This article therefore links to course materials and has no corresponding lecture player. Géron/Karpathy videos are supplementary readings; the CMU-Qatar NumPy recording is a separate Recitation 0, not the lecture covered here.

Official sources:

• CMU 07-280 Spring 2026 官方課表與教材

Checked on 2026-10-10.

Official materials and reading scope

I read the complete Linear Regression lecture notes, Optimization and Linear Regression pre-reading, Recitation 4 and solution, and checked HW4. No lecture recording is public.

Update Log

• 2026-10-10: Added explicit video status and checked recording sources and access notes.
• 2026-10-10: Rechecked video status. Re-verified the live official course pages and public video sources; no public recording for this lecture was found, so the status stands.

References

• CMU 07-280 Spring 2026 Linear Regression lecture notes
• 07-280 Optimization and Linear Regression pre-reading
• 07-280 Spring 2026 Recitation 4
• Recitation 4 solution
• 07-280 Spring 2026 Homework 4