Backprop is not new math; it computes the part of the chain rule that every layer shares exactly once. For a linear layer the forward pass multiplies by W, the backward pass multiplies by W transposed, and the weight gradient is 'input times the gradient coming back'. Gradients add at a branch, so shared parameters add their gradients too. A hand-computed update on a 2-2-1 network at the end takes the loss from 0.0175 to 0.0076.
The lecture starts from one theorem: if a scalar function can be computed with N basic operations, its gradient can also be computed in O(N) time. The method is to read the chain rule as "given ∂J/∂u, you can compute ∂J/∂z, using only information about the module in between", then derive a backward function for matrix multiplication and for the activation function: Wᵀv, vzᵀ, and σ′(z)⊙v. This article also works a forward and backward pass by hand on a small 2-2-1 network.
L05 follows a small sentiment classification network throughout. It expresses computation as a graph, evaluates it in topological order, sends gradients back with the chain rule and vector-Jacobian products, and then takes apart TensorFlow v1's placeholder, variable, operation, and session. One slide is labeled "important for HW2".
The second 11-868 assignment has three parts: autodiff's topological_sort and backpropagate (40 points), a Linear layer and MLP network (30), and binary cross entropy plus the training loop (30). You then train a sentiment classifier on SST-2 with GloVe embeddings and must reach 75% validation accuracy. The default backend is the CUDA kernels from Assignment 1, and the repo merged small Fall 2026 fixes on 2026-09-02.
The first half of L4 replaces the linear classifier f = Wx with a two-layer network f = W₂ max(0, W₁x) and shows that dropping the max activation collapses it back into a linear classifier. The second half answers how to compute gradients once the network gets deep: draw the function as a computational graph, and each node only needs its own local gradient multiplied by the upstream gradient coming back from later nodes. Add distributes, mul swaps, max routes, copy sums, and those four patterns let you trace any network. The lecture ends with matrices: dL/dx always has the same shape as x, so never build the Jacobian.
Training a model means answering three questions. What is the model: layers of neurons, each computing z = Wa + b followed by a non-linearity; one neuron draws only one boundary, so it cannot compute XOR, and stacking layers can. What does a good function mean: define a loss, and smaller is better. How to pick the best: gradient descent takes a step along the slope where it stands, and practice uses mini-batch SGD. With millions of parameters, gradients come from backpropagation: a forward pass stores every layer's output, a backward pass computes δ from the output layer back, and their product is each weight's gradient. The guide works each of these through with numbers.
A linear aggregation of perceptrons can implement AND and OR but not XOR; one more layer can, and that is the multi-layer perceptron. This lecture replaces the sign in the middle with tanh, puts linear regression at the output layer, and derives backpropagation from the chain rule: compute every layer's output x forward, then compute δ backwards from the output layer, and the gradient of each weight is the previous layer's x times this layer's δ. It ends with two troubles in training: E_in is non-convex, so the initial weights should be small and random; d_VC is roughly O(VD), so weight elimination and early stopping are needed against overfitting.
HW3 has two halves. The four written problems run from Newton's method for logistic regression and a convergence analysis of coordinate descent, through backprop, VJPs, and implicit differentiation, to an information-bottleneck view of what deep networks compress. The notebook has you build a BearTensor computation graph in NumPy, topological-sort backprop, and SGD/Momentum/Adam, then train a red-wine quality regressor with it, plus an optional Muon optimizer. Problems and notebook are public; official solutions and hidden tests are not.
Lec 17 uses XOR to show that a linear model, and even a stack of linear layers, cannot learn a nonlinear boundary, while one ReLU layer can. The universal approximation theorem guarantees that a network exists but not how to find its weights or how wide it must be. Lec 18 turns the chain rule into backpropagation on a computation graph: gradients from multiple paths add up, and the cost is linear in the number of parameters, versus quadratic for finite differences. Discussion 8 has you prove a GD convergence rate and 1-D ReLU universal approximation yourself.
CS181 Spring 2026 HW3 (due Mar 23) has three problems worth 100 points: unpack polynomial and RBF kernels into feature maps and go from ridge to dual coefficients α and the support-vector intuition; hand-derive backprop for a two-layer sigmoid network; and, for half the grade, train ResNets on Fashion-MNIST, measure your own scaling law, and use C≈6ND to split a fixed compute budget between model and data.
Week 5 is about neural networks: a unit multiplies its inputs by weights, adds a bias and applies an activation function; gradient descent adjusts the weights one small step at a time, and backpropagation passes the error back so hidden layers can be trained; for images, convolution and pooling come first and an ordinary network follows, which is a CNN; for sequences, an RNN feeds its output back into itself. With time links for every section, plus the approach, common mistakes and checks for the Traffic project.
A model uses loss to know how wrong it is and gradients to know which direction to adjust. Gradient descent repeats three things: compute loss, compute gradients, update parameters. The learning rate controls step size — too large and you overshoot, too small and training takes forever.
Lecture 12 treats a network as a computation graph: the forward pass stores intermediates, the backward pass propagates upstream gradients, and local linear, activation, and softmax rules compute every parameter gradient efficiently.
Lecture 3 decomposes neural-network training into computation graphs, local derivatives, and the chain rule: the forward pass computes a result; backprop accumulates gradients from the output so every parameter knows how to move.