Lecture 7 first carries the training recipe for linear models over to nonlinear models unchanged: one loss per example, take the average, walk downhill with SGD. Two questions remain: what the model looks like, and how to compute the gradient. The second half answers the first. One neuron is ReLU(wᵀx+b); stacking m neurons into a matrix gives one layer, ReLU(Wx+b), with md+m parameters; above that come the residual block z+σ(MM(σ(MM(z)))) and LayerNorm and RMSNorm. Computing the gradient is left to Lecture 8.
The lecture starts from one theorem: if a scalar function can be computed with N basic operations, its gradient can also be computed in O(N) time. The method is to read the chain rule as "given ∂J/∂u, you can compute ∂J/∂z, using only information about the module in between", then derive a backward function for matrix multiplication and for the activation function: Wᵀv, vzᵀ, and σ′(z)⊙v. This article also works a forward and backward pass by hand on a small 2-2-1 network.
Interview follow-ups usually land on mechanisms: why divide by √d_k, how AdamW differs from Adam plus L2, why a high AUC still isn't shippable. This post walks through concept, mechanism and how to answer, then ends with 3 runnable numpy exercises.
CS231N Assignment 1 is worth 12% of the grade and was due April 16, 2026. All five Colab notebooks are hand-written numpy on CIFAR-10. Q1 kNN asks for distance computations with two loops, one loop, and no loops. Q2 Softmax goes from naive to vectorized to SGD. Q3 assembles affine, ReLU, and softmax into a two-layer network. Q4 switches to HOG and color-histogram features. Q5 generalizes to any depth and implements Momentum, RMSProp, and Adam. The 65 KB starter code is public; the Gradescope grading isn't. This post covers structure and goals only, not solutions.
The first half of L4 replaces the linear classifier f = Wx with a two-layer network f = W₂ max(0, W₁x) and shows that dropping the max activation collapses it back into a linear classifier. The second half answers how to compute gradients once the network gets deep: draw the function as a computational graph, and each node only needs its own local gradient multiplied by the upstream gradient coming back from later nodes. Add distributes, mul swaps, max routes, copy sums, and those four patterns let you trace any network. The lecture ends with matrices: dL/dx always has the same shape as x, so never build the Jacobian.
Lecture 2 opens up last week's "dopey AI robot." Inputs and outputs must become numbers (tensors). Classification uses one-hot labels and softmax to turn scores into probabilities. A neural network is neurons stacked layer by layer, and training means pushing the loss down with gradient descent. The lecture ends by building a first fully connected network on MNIST in Keras and wiring it to a Gradio sketchpad. Homework 2 asks you to design your own DNN, with one hard rule: it can't have three layers. The Chang Gung satellite rubric wants a screenshot of the parameters with the best validation accuracy and encourages keeping failed attempts.
Training a model means answering three questions. What is the model: layers of neurons, each computing z = Wa + b followed by a non-linearity; one neuron draws only one boundary, so it cannot compute XOR, and stacking layers can. What does a good function mean: define a loss, and smaller is better. How to pick the best: gradient descent takes a step along the slope where it stands, and practice uses mini-batch SGD. With millions of parameters, gradients come from backpropagation: a forward pass stores every layer's output, a backward pass computes δ from the output layer back, and their product is each weight's gradient. The guide works each of these through with numbers.
A linear aggregation of perceptrons can implement AND and OR but not XOR; one more layer can, and that is the multi-layer perceptron. This lecture replaces the sign in the middle with tanh, puts linear regression at the output layer, and derives backpropagation from the chain rule: compute every layer's output x forward, then compute δ backwards from the output layer, and the gradient of each weight is the previous layer's x times this layer's δ. It ends with two troubles in training: E_in is non-convex, so the initial weights should be small and random; d_VC is roughly O(VD), so weight elimination and early stopping are needed against overfitting.
Lec 17 uses XOR to show that a linear model, and even a stack of linear layers, cannot learn a nonlinear boundary, while one ReLU layer can. The universal approximation theorem guarantees that a network exists but not how to find its weights or how wide it must be. Lec 18 turns the chain rule into backpropagation on a computation graph: gradients from multiple paths add up, and the cost is linear in the number of parameters, versus quadratic for finite differences. Discussion 8 has you prove a GD convergence rate and 1-D ReLU universal approximation yourself.
Lec 19 wraps up backprop, then tackles how to keep gradients flowing: all-zero initialization makes every unit learn the same thing, so use small random values (He init for ReLU), and batch norm normalizes pre-activations with mini-batch means and variances. The second half introduces CNNs: local connectivity plus weight sharing lets one feature detector scan the whole image. Lec 20 finishes pooling, receptive fields, and CNN training, then covers early stopping, dropout, and double descent, which breaks the classic bias-variance picture.
CS181 Spring 2026 HW3 (due Mar 23) has three problems worth 100 points: unpack polynomial and RBF kernels into feature maps and go from ridge to dual coefficients α and the support-vector intuition; hand-derive backprop for a two-layer sigmoid network; and, for half the grade, train ResNets on Fashion-MNIST, measure your own scaling law, and use C≈6ND to split a fixed compute budget between model and data.
Week 5 is about neural networks: a unit multiplies its inputs by weights, adds a bias and applies an activation function; gradient descent adjusts the weights one small step at a time, and backpropagation passes the error back so hidden layers can be trained; for images, convolution and pooling come first and an ordinary network follows, which is a CNN; for sequences, an RNN feeds its output back into itself. With time links for every section, plus the approach, common mistakes and checks for the Traffic project.
Spring 2026 Lecture 1 focuses on neurons, perceptrons, connectionism, and the problem framing of deep learning. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 2 focuses on Boolean functions, how network depth and width affect representational capacity (including a multi-input XOR example), and the hand-off at the end to the next lecture's training problem. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 3 focuses on how the learning problem is set up (representing inputs and outputs as numbers), the perceptron learning rule and why it does not carry over to multi-layer networks, differentiable divergences, and empirical risk minimization (the loss). This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 4 focuses on the definitions of derivative and gradient, extrema and second derivatives, gradient descent as repeated steps against the gradient, and the network, input/output representations, and divergence that must be defined before training. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 5 focuses on layered-network notation, the forward pass, the chain rule and Jacobians (including vector activations), and how the backward pass computes weight and bias derivatives layer by layer. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 6 focuses on convergence of gradient descent (step-size choice on convex and quadratic functions, differences between dimensions), local minima and saddle points, decaying or adaptive learning rates, R-prop, and momentum with Nesterov accelerated gradient. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 7 focuses on incremental SGD and mini-batch updates and trend-based updates such as RMSprop and momentum (second-order methods are only in the slides, not in the recording). This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 8 focuses on the choice of divergence (loss), BatchNorm, dropout and other regularization methods, and training tricks such as early stopping (AdaGrad and Adam are not in this recording). This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 9 focuses on why we scan the input, distributing the scan over several layers, handling deformation with max pooling, and the CNN / time delay neural network structure. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 10 focuses on where the CNN comes from (Hubel and Wiesel, Fukushima's Neocognitron), the operations of convolution and pooling layers, and design choices such as filter size, stride and number of channels. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 11 focuses on a recap of CNNs and backpropagation through convolution and pooling layers. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 12 focuses on additional backpropagation rules (strides, upsampling and downsampling), invariance to transformations other than shifts, and a quick history of CNNs with the major architectures. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 13 focuses on why sequences need their own models, state and memory, Jordan/Elman-style recurrent networks, backpropagation through time, and bidirectional networks. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 14 focuses on backpropagation through time, gradient stability, and LSTM-style gated memory. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 15 focuses on the alignment problem when training sequence models (training with explicit or re-estimated alignments, and the risk of local optima); CTC is only introduced at the very end, with details left to the next lecture. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 16 focuses on blank symbols and the merging of adjacent repeated symbols, CTC's forward (alpha) probability computation, and pruned approximate decoding (beam search is described only briefly near the end). This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 17 focuses on next-word-prediction language models, conditional translation models, and beam-search decoding. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 18 focuses on sequence-to-sequence attention (queries, keys, values, and softmax-weighted averaging), a mention of multi-head attention, and a brief introduction to the Transformer. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 19 focuses on encoder/decoder structures, masks, residual paths, and architecture variants. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 20 focuses on scaled autoregressive models, pre-training and scaling laws, and post-training (including improving the model from its own sampled answers and feedback). This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 21 focuses on what neural networks actually represent, the reconstruction objective (autoencoders), and the link to PCA (dimensionality reduction). This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 22 focuses on latent variables, the ELBO, the KL term, and the reparameterization trick. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 23 focuses on forward noising, reverse denoising, score or noise prediction, and sampling. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 24 focuses on the generator, discriminator, minimax objective, and training instability. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 25 focuses on message passing, aggregation, node representations, and permutation symmetry. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 26 focuses on states, actions, rewards, returns, values, and policy learning. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 27 focuses on associative memory, energy functions, fixed points, and pattern retrieval. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 28 focuses on energy-based probability models (the Gibbs distribution and its normalizing constant Z), stochastic neurons, and how to train such probabilistic networks. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Spring 2026 publishes official slides and YouTube recordings for all 28 content lectures, plus extensive bootcamps and recitations. Its HW1–HW4 specifications, starters, and evaluation still depend on Autolab, Piazza, and Kaggle.