The representer theorem turns ridge regression into a problem in β, which gives kernel ridge regression the analytic solution β = (λI + K)⁻¹y, but almost none of the N values βₙ are 0. Replace the squared error with the ε-insensitive tube error, write the problem as a QP along the soft-margin SVM route, take the dual, and you get support vector regression (SVR): the dual has 2N variables, and every point strictly inside the tube has βₙ equal to 0.
The ξₙ of the soft-margin SVM can be written directly as max(1 − yₙ(wᵀzₙ + b), 0), which turns the whole problem into ½wᵀw plus C times the sum of hinge errors: an L2-regularized model in which C corresponds to 1/(2λ). The hinge error and logistic regression's cross-entropy are both convex upper bounds of the 0/1 error, so the SVM approximates L2-regularized logistic regression. There are two routes to probability outputs: Platt's two-level learning learns only two extra numbers, A and B; the representer theorem guarantees that the optimal w is a linear combination of the zₙ, so kernel logistic regression can be done directly in the Z space, at the price that the N coefficients βₙ are usually all nonzero.
The dual SVM's variable count no longer depends on the feature dimension d̃, but every entry of the Q matrix is still one d̃-dimensional inner product. This lecture merges "transform + inner product" into a single kernel function K(x, x′); for the second-order transform the inner product takes only O(d). Once training, computing b, and prediction all use K, the SVM never touches the Z space. Pushed to the limit, the same move gives the Gaussian kernel, which corresponds to an infinite-dimensional transform, at the price that too large a γ still overfits.
CS181 Spring 2026 HW3 (due Mar 23) has three problems worth 100 points: unpack polynomial and RBF kernels into feature maps and go from ridge to dual coefficients α and the support-vector intuition; hand-derive backprop for a two-layer sigmoid network; and, for half the grade, train ResNets on Fashion-MNIST, measure your own scaling law, and use C≈6ND to split a fixed compute budget between model and data.