When no straight line separates the data, you do not need to redesign the algorithm: use a feature transform Φ to turn x into z, run a linear model in Z-space as usual, and the line in Z-space is a quadratic or higher-order curve in X-space. The Q-th order polynomial transform raises the dimension to d̃ = C(Q+d, d) − 1, and computation, storage, and the VC dimension (at most d̃+1) all grow with it, so the lecture concludes that you should try the linear model first and move to higher orders only when it is not enough.
Stepping back from a 10th-order polynomial to a 2nd-order one is the same as adding a constraint to the 10th-order problem. Once the constraint is softened to ‖w‖² ≤ C, the negative gradient of E_in at the optimal solution must be parallel to w, which gives w_REG = (ZᵀZ + λI)⁻¹Zᵀy and is the same as minimizing E_aug = E_in + (λ/N)wᵀw. That is weight decay. A larger λ means a smaller equivalent C and a smaller effective VC dimension d_EFF; switching to L1 gives sparse solutions instead.
The last Foundations lecture teaches no new algorithm. It covers three situations in which the guarantees of the first fifteen lectures stop holding. A simple model rarely separates randomly labeled data by accident: with 10 points and random labels, a decision stump separates them with probability only 20/1024, so a successful separation means something. The VC guarantee fails when training and testing do not share one distribution, and once data has affected any step of the learning process it can no longer assess the outcome. The lecture closes with Power of Three, which folds the whole course into five groups of three.
The primal hard-margin SVM has d̃+1 variables, so it stops being solvable once the feature dimension gets large. This lecture uses Lagrange duality to trade it for a quadratic program with only N variables and N+1 constraints, then uses the KKT conditions to recover w and b from α; only the support vectors, the points with α greater than 0, affect the answer. The dual's matrix still contains d̃-dimensional inner products, so the feature dimension is not really gone until the next lecture.
Hard-margin SVM insists on separating the data perfectly, so it fits noise. This lecture gives every point a violation ξₙ and uses the parameter C to trade margin width against total violation, which yields soft-margin SVM; its dual differs from the hard-margin dual only in that αₙ gains the upper bound C. The value of αₙ sorts the data into non-support vectors, free SVs, and bounded SVs, and b is computed from a free SV; C and γ are chosen by cross-validation, and the fraction of support vectors #SV/N is an upper bound on the leave-one-out error.
The representer theorem turns ridge regression into a problem in β, which gives kernel ridge regression the analytic solution β = (λI + K)⁻¹y, but almost none of the N values βₙ are 0. Replace the squared error with the ε-insensitive tube error, write the problem as a QP along the soft-margin SVM route, take the dual, and you get support vector regression (SVR): the dual has 2N variables, and every point strictly inside the tube has βₙ equal to 0.
This lecture re-reads bagging's bootstrap as putting a weight on every example, then deliberately tunes the weights so that the previous round's gₜ has a weighted error rate of exactly 1/2, which forces the next round to learn a different hypothesis. The recipe multiplies incorrect examples by ◆ₜ = √((1−εₜ)/εₜ), divides correct ones by it, and uses αₜ = ln ◆ₜ as the vote weight in a linear vote. That is AdaBoost. As long as the base algorithm is a little better than random guessing in every round, E_in drops to 0 within O(log N) rounds; with decision stumps, three hand-computed rounds on six examples are enough to see it.
A random forest is bagging applied to fully-grown C&RT trees, with each branch looking only at a few randomly projected features. When bootstrap draws N times, each example has about a 1/e ≈ 36.8% chance of not being used by a given tree. These out-of-bag examples give E_oob directly, a validation that needs no held-out data and no re-training; add the permutation test and you get an importance for every feature. The price is that the result wobbles with the random seed, so you keep adding trees until G is stable.
The sum of AdaBoost's weights in each round is exactly the sum of the exponential error exp(−ys). From that, this lecture derives that the gₜ found by the base algorithm is an approximate functional gradient direction, and the vote αₜ is the best step size along that direction. Swap the exponential error for squared error and the same derivation becomes 'in each round, run regression on the residuals yₙ − sₙ, then set the step size with one-variable linear regression'; with a weakened C&RT as the base model, that is GBDT.
Once a network is stacked deep, each layer only has to do a little, but structure, model complexity, optimization, and computation all get harder. This lecture picks two of those, initialization and regularization: pre-train the weights layer by layer with a d—d̃—d autoencoder so that each layer first learns an information-preserving encoding; train on inputs with artificial noise added, as a form of regularization; and finally make the autoencoder linear and derive that the optimal W is the d̃ top eigenvectors of XᵀX, which becomes PCA once the mean is subtracted from the data.
The only input a recommender system gets is a user ID. Apply binary vector encoding to the ID and feed it to an N-d̃-M linear network with the tanh removed, and the hypothesis is h(x) = WᵀVx, which is the same as writing the rating matrix as R ≈ VᵀW. There are two ways to learn V and W: alternating least squares solves M+N linear regressions per round, and SGD uses one rating at a time to update the vectors of one user and one movie. The lecture closes by putting boosting, neural networks, RBF networks, matrix factorization, and k nearest neighbor on one map of extraction models.
CS230 is the deep learning course Andrew Ng and Kian Katanforoosh teach at Stanford, run as a flipped classroom: the foundations of neural networks, optimization, CNNs, sequence models and Transformers live in five online Coursera courses, and one in-person lecture a week covers project method, adversarial attacks and generative models, reinforcement learning, LLM applications and interpretability. The nine in-person lectures, most of their slides and two practice midterms with solutions are public; the Stanford-only Coursera sessions, the section recordings and project mentoring are not.
The opening lecture of CS229 Spring 2026 runs only 37 minutes and has no derivations. It defines machine learning with Tom Mitchell's experience E, task T, and performance measure P, then splits the whole course into supervised, unsupervised, and reinforcement learning. The three differ in whether the data comes with answers and whether the model collected the data itself. Large language models show up in all three categories at once.
Lecture 2 breaks supervised learning into three pieces: a hypothesis h, the least-squares cost J(θ), and an algorithm that drives J down. Gradient descent sweeps all n examples on every step; minibatch SGD updates after looking at a small batch, and pays for it by bouncing around near the optimum. Linear regression also has a closed-form solution, θ = (XᵀX)⁻¹Xᵀy, provided XᵀX is invertible.
Assume the errors are zero-mean, IID Gaussian noise, take the log of the likelihood, and maximum likelihood turns out to be last lecture's least squares. Run the same recipe with 0/1 labels and a sigmoid and you get logistic regression, whose gradient is still "error times input". The lecture closes by comparing Newton's method with SGD: far fewer steps, but each step costs n·d² + d³.
If the distribution of y can be written as b(y)·exp(ηᵀT(y) − a(η)), one derivative of a(η) gives the expectation and two give the variance. Set η = θᵀx, and linear regression, logistic regression, and softmax all share one update rule: θ ← θ + α(y − h(x))x. The second half of the lecture uses a four-class example to cover softmax, cross entropy, and label smoothing.
Lecture 5 turns classification around: instead of learning p(y|x) directly, it learns p(x|y) and p(y). GDA assumes the two classes are Gaussians with a shared covariance. Maximum likelihood then takes nothing more than counting proportions and averaging, the decision boundary comes out as a straight line, and the posterior is exactly a sigmoid. Naive Bayes uses conditional independence to squeeze 2^d parameters down to 2d+1.
This lecture splits test error into three terms: noise σ², squared bias, and variance. It then uses the closed-form solution of ridge regression to show how regularization moves each eigenvalue from λ to λ+ρ, trading a little bias for a large cut in variance. The second half covers double descent, ImageNet-V2, and choosing hyperparameters with a dev set, k-fold cross-validation, and successive halving.
Lecture 7 first carries the training recipe for linear models over to nonlinear models unchanged: one loss per example, take the average, walk downhill with SGD. Two questions remain: what the model looks like, and how to compute the gradient. The second half answers the first. One neuron is ReLU(wᵀx+b); stacking m neurons into a matrix gives one layer, ReLU(Wx+b), with md+m parameters; above that come the residual block z+σ(MM(σ(MM(z)))) and LayerNorm and RMSNorm. Computing the gradient is left to Lecture 8.
The lecture starts from one theorem: if a scalar function can be computed with N basic operations, its gradient can also be computed in O(N) time. The method is to read the chain rule as "given ∂J/∂u, you can compute ∂J/∂z, using only information about the module in between", then derive a backward function for matrix multiplication and for the activation function: Wᵀv, vzᵀ, and σ′(z)⊙v. This article also works a forward and backward pass by hand on a small 2-2-1 network.