The only input a recommender system gets is a user ID. Apply binary vector encoding to the ID and feed it to an N-d̃-M linear network with the tanh removed, and the hypothesis is h(x) = WᵀVx, which is the same as writing the rating matrix as R ≈ VᵀW. There are two ways to learn V and W: alternating least squares solves M+N linear regressions per round, and SGD uses one rating at a time to update the vectors of one user and one movie. The lecture closes by putting boosting, neural networks, RBF networks, matrix factorization, and k nearest neighbor on one map of extraction models.
Backprop is not new math; it computes the part of the chain rule that every layer shares exactly once. For a linear layer the forward pass multiplies by W, the backward pass multiplies by W transposed, and the weight gradient is 'input times the gradient coming back'. Gradients add at a branch, so shared parameters add their gradients too. A hand-computed update on a 2-2-1 network at the end takes the loss from 0.0175 to 0.0076.
Lecture 7 first carries the training recipe for linear models over to nonlinear models unchanged: one loss per example, take the average, walk downhill with SGD. Two questions remain: what the model looks like, and how to compute the gradient. The second half answers the first. One neuron is ReLU(wᵀx+b); stacking m neurons into a matrix gives one layer, ReLU(Wx+b), with md+m parameters; above that come the residual block z+σ(MM(σ(MM(z)))) and LayerNorm and RMSNorm. Computing the gradient is left to Lecture 8.
PLA, linear regression, and logistic regression all start by computing the score s = wᵀx, and all three errors can be written as functions of ys. The squared error and the cross-entropy rescaled to log₂ both sit above the 0/1 error, so making the regression E_in small also gives a guarantee on the classification E_out. Next, replacing the average over N examples in the gradient with one randomly drawn example gives SGD at O(1) per step. Finally, OVA (K classifiers, take the highest score) and OVO (K(K−1)/2 classifiers, vote) assemble binary classifiers into a multiclass classifier.