Once a network is stacked deep, each layer only has to do a little, but structure, model complexity, optimization, and computation all get harder. This lecture picks two of those, initialization and regularization: pre-train the weights layer by layer with a d—d̃—d autoencoder so that each layer first learns an information-preserving encoding; train on inputs with artificial noise added, as a form of regularization; and finally make the autoencoder linear and derive that the optimal W is the d̃ top eigenvectors of XᵀX, which becomes PCA once the mean is subtracted from the data.
CS230 is the deep learning course Andrew Ng and Kian Katanforoosh teach at Stanford, run as a flipped classroom: the foundations of neural networks, optimization, CNNs, sequence models and Transformers live in five online Coursera courses, and one in-person lecture a week covers project method, adversarial attacks and generative models, reinforcement learning, LLM applications and interpretability. The nine in-person lectures, most of their slides and two practice midterms with solutions are public; the Stanford-only Coursera sessions, the section recordings and project mentoring are not.
Today's Deep Learning & NLP rotation covers four core concepts: self-attention scores relevance with a Query/Key dot product and uses it to weight the Value vectors, at the cost of quadratic compute in sequence length; causal masks and padding masks solve two unrelated problems — one controls whether a position can see the future, the other controls whether padding should be attended to at all; subword tokenization (BPE/WordPiece) solves out-of-vocabulary words by falling back to known sub-word pieces, though tokenizer fertility still varies by language; and RNNs vs. Transformers boils down to a fixed-size hidden state that has to carry information forward step by step, versus attention that can reach any position directly. The practice question is a classic architecture question: GPT only uses the Transformer's decoder, relying on causal attention for autoregressive generation — the breakdown threads 'why half the architecture is enough' through to 'how KV caching makes inference fast' into one complete reasoning chain.
Backprop is not new math; it computes the part of the chain rule that every layer shares exactly once. For a linear layer the forward pass multiplies by W, the backward pass multiplies by W transposed, and the weight gradient is 'input times the gradient coming back'. Gradients add at a branch, so shared parameters add their gradients too. A hand-computed update on a 2-2-1 network at the end takes the loss from 0.0175 to 0.0076.
Neither parameter count nor VC dimension explains generalization in deep nets: a network can memorize random labels to 100%, which makes the VC bound vacuous, and past the interpolation threshold test error falls a second time as parameters grow. Isola's account is that there are pressures beyond fitting the data, and he considers architectural symmetry and compositionality the most important. The question is still open.
A generative model plus its latent space works as data you can operate on: moving along a latent direction yields counterfactual images, and a handful of manual labels expands into a whole segmentation dataset (one labeled GAN image is worth about 100 ordinary labeled images). But representations trained on generated data still trail real data (42.6% against 43.9% in the numbers cited in class), and rare classes are exactly where generators are least reliable.
Test loss follows an approximate power law in data, parameters and compute, but the exponents are small: Kaplan et al. measure a data exponent of about 0.095, so doubling the data lowers the loss by only 6% and halving it takes over 1,400 times more. Two papers disagree on how to split compute: Kaplan gives parameters ∝ C^0.73 and data ∝ C^0.27, Chinchilla about half each. The public recording stops at the second finding; everything after that is written here from the slides, without the lecturer's spoken explanation.
A language model does one thing: given the preceding text, output a distribution over the next word. At large enough scale, the way to control it becomes changing the input: a few examples (in-context learning), or asking it to write its reasoning first (chain-of-thought; the example cited in class is one sentence, Let's think step by step, taking an arithmetic task from 18% to 80%). But following instructions and saying 'I don't know' do not come from next-word prediction alone; instruction tuning and RLHF add them.
Gradient descent silently assumes the weight space is Euclidean, and for neural networks it mostly is not. Bernstein's approach picks a norm for each layer (a shape-rescaled spectral norm for linear layers) and computes a norm for the whole network from two combination rules; normalize updates in it and the optimal learning rate stops drifting with width and depth. Push the same idea to duality maps and a linear layer's update becomes 'set every singular value of the gradient to 1', which is also what the nanoGPT training speed record of the time used.
A trained model is not limited to one forward pass. Greedy sampling does not guarantee the sequence the model itself considers most likely (in this post's example it gets 0.30 where the best is 0.36), so sampling several times and keeping the best already helps; swap the scoring criterion for a verifier and you can pick answers that are actually correct. Feed the good answers found by search back into training and you have the loop behind AlphaGo and STaR, which Isola speculates is also how o1 works.
Lecture 7 first carries the training recipe for linear models over to nonlinear models unchanged: one loss per example, take the average, walk downhill with SGD. Two questions remain: what the model looks like, and how to compute the gradient. The second half answers the first. One neuron is ReLU(wᵀx+b); stacking m neurons into a matrix gives one layer, ReLU(Wx+b), with md+m parameters; above that come the residual block z+σ(MM(σ(MM(z)))) and LayerNorm and RMSNorm. Computing the gradient is left to Lecture 8.
The lecture starts from one theorem: if a scalar function can be computed with N basic operations, its gradient can also be computed in O(N) time. The method is to read the chain rule as "given ∂J/∂u, you can compute ∂J/∂z, using only information about the module in between", then derive a backward function for matrix multiplication and for the activation function: Wᵀv, vzᵀ, and σ′(z)⊙v. This article also works a forward and backward pass by hand on a small 2-2-1 network.
Today's Deep Learning & NLP rotation covers five core concepts: why Self-Attention's scaled dot-product divides by sqrt(d_k); how tokenizer fertility (tokens per word) decides a language's cost and effective context length; why BERT's attention mask and MLM loss label are two independent mechanisms that interviewees often conflate; the trade-offs between CNN and RNN; and the fine-tuning spectrum from full fine-tuning to LoRA to continued pretraining. The practice question comes from a representative Sarvam AI interview question (compiled by the AI-Engineer-Interview-Questions project from public information, not leaked): explain why tokenization is the first performance bottleneck for multilingual LLMs, and how a low-fertility tokenizer like Sarvam-1's changes the underlying cost structure — the breakdown threads 'how fertility is defined' through to 'which three downstream cost lines it determines' into one complete reasoning chain.
Interview follow-ups usually land on mechanisms: why divide by √d_k, how AdamW differs from Adam plus L2, why a high AUC still isn't shippable. This post walks through concept, mechanism and how to answer, then ends with 3 runnable numpy exercises.
L05 follows a small sentiment classification network throughout. It expresses computation as a graph, evaluates it in topological order, sends gradients back with the chain rule and vector-Jacobian products, and then takes apart TensorFlow v1's placeholder, variable, operation, and session. One slide is labeled "important for HW2".
The second 11-868 assignment has three parts: autodiff's topological_sort and backpropagate (40 points), a Linear layer and MLP network (30), and binary cross entropy plus the training loop (30). You then train a sentiment classifier on SST-2 with GloVe embeddings and must reach 75% validation accuracy. The default backend is the CUDA kernels from Assignment 1, and the repo merged small Fall 2026 fixes on 2026-09-02.
Assignment 2 of CS231N Spring 2026 is worth 18% of the grade. Across five notebooks you hand-write BatchNorm/LayerNorm, dropout, and the forward and backward passes for convolution and pooling, then learn PyTorch at three levels of abstraction, and finish with RNN image captioning on COCO in PyTorch. Q4 is the turning point: through Q3 you derive every gradient yourself, and from Q5 on autograd takes over while a numerical gradient check confirms it. The official slides warn that this is the longest of the three assignments.
A two-layer network flattens a 32×32×3 image into a 3072-dimensional vector, and the spatial structure is gone. CS231N Lecture 5 answers with two layers. A convolution layer slides small filters across the image and reuses the same weights at every position. A pooling layer downsamples and has no learnable parameters. Both are translation equivariant. One formula gives every layer's output size: (W − K + 2P) / S + 1.
CS231N is Stanford's deep learning course for computer vision. For Spring 2026, slides for 16 lectures, all three assignment pages with starter code, the course notes, and the project spec are public, so this series rates it A3 (enough to self-study). There are three gaps: the 2026 recordings are Canvas-only, L17 and L18 have no slides, and the midterm is not public. You can pair the 2026 slides and assignments with the 2025 YouTube recordings and follow the official calendar over 10 weeks.
L2 starts from one question: a computer sees a grid of numbers between 0 and 255, so how does it recognize a cat? Hand-written rules don't scale, so the course switches to a data-driven approach: collect data, train, evaluate on new images. The first classifier, kNN, teaches how to split train/val/test, but pixel distances carry no meaning. The second, the linear classifier f(x,W)=Wx+b, can be read three ways (algebraic, visual as templates, geometric as hyperplanes). Softmax turns its scores into probabilities, and the loss is the negative log probability of the correct class.
The first CS231N lecture of 2026 comes in two slide decks. The first tells the history of vision and deep learning on a single timeline: Hubel & Wiesel's cat experiments, Marr's stages of visual representation, then the Neocognitron, backprop, and LeNet, until ImageNet and AlexNet join the two threads. The second covers the course map, grading, and rules, and moves every assignment onto Colab. After this lecture you'll know which gap each of the remaining 17 lectures fills.
The first half of L4 replaces the linear classifier f = Wx with a two-layer network f = W₂ max(0, W₁x) and shows that dropping the max activation collapses it back into a linear classifier. The second half answers how to compute gradients once the network gets deep: draw the function as a computational graph, and each node only needs its own local gradient multiplied by the upstream gradient coming back from later nodes. Add distributes, mul swaps, max routes, copy sums, and those four patterns let you trace any network. The lecture ends with matrices: dL/dx always has the same shape as x, so never build the Jacobian.
An RNN updates one hidden state at every time step with the same weights, so it can handle sequences of any length. CS231N Lecture 7 starts by hand-building an RNN that detects repeated 1s. It then covers character-level language models and feeding CNN features into an RNN for image captioning. Gradient flow explains why vanilla RNNs are hard to train: clip gradients to stop them exploding, and change the architecture (the LSTM) to stop them vanishing. The lecture ends by calling state space models like Mamba "modern RNNs."
L2 gave us a score function and a loss. L3 answers how to find a good W. The first half covers regularization: add λR(W) next to the data loss so the model does not fit the training data too well. The second half is a lineage of optimizers. SGD zigzags in narrow valleys, Momentum builds up velocity, RMSProp scales each dimension's step, Adam combines the two and adds bias correction, and AdamW moves weight decay outside the moment estimates. The slides close with practical advice: Adam(W) is a good default in many cases, and SGD+Momentum can do better but needs more tuning of the learning rate and schedule.
The 2026 slides for CS231N Lecture 6 are titled "Training CNNs and CNN Architectures" and split into how to build and how to train. Only two architectures get case studies. VGG shows that three 3×3 convs are deeper and cheaper than one 7×7. ResNet lets layers learn the residual F(x) = H(x) − x, which fixes an optimization problem where deeper plain nets had worse training error. The most practical takeaway is transfer learning: with fewer than about a million images, start from a model pretrained on a large dataset.
The first two lectures of 6.5940 show that the problem exists, then hand you the rulers. L1 plots model parameter counts growing much faster than GPU memory, and contrasts 80GB on a cloud GPU with 320kB on a microcontroller. L2 splits efficiency metrics into memory metrics (#parameters, model size, peak activations) and compute metrics (MAC, FLOP, OP). AlexNet has 61M parameters and 724M MACs, and on a microcontroller the thing that runs out first is usually activation memory, not parameters. Lab 0 introduces a VGG variant on CIFAR-10 (9.2M parameters, 606M MACs) that later labs build on.
Lecture 7 has three parts. It first reviews fully connected, convolution, grouped, depthwise, and 1×1 convolution layers through their MAC formulas. It then takes apart how the ResNet bottleneck, ResNeXt, MobileNet, MobileNetV2, ShuffleNet, and the Transformer each save compute; the bottleneck, for example, needs 8.5× fewer MACs than a plain 3×3 convolution over 2048 channels. The last part is NAS: search spaces are either cell-level or network-level (depth, resolution, width, kernel size, topology), and there are five search strategies: grid, random, reinforcement learning, gradient descent, and evolution. One arithmetic exercise on the slides shows that the NASNet cell space already holds 3.2×10¹¹ candidates at M=5, N=2, B=5.
Pruning removes unimportant weights or neurons from a neural network. The goal is written as minimizing loss subject to at most N nonzero weights. Lecture 3 of 6.5940 handles two of the decisions involved. First, granularity: from fine-grained pruning, which can remove any element, to channel pruning, which removes whole channels. The more regular the pattern, the easier it is to speed up on existing hardware, and the less you can remove. In between, 2:4 sparsity gives up to 2× speedup on NVIDIA Ampere GPUs. Second, criteria: look at weight magnitude, Batch Norm scaling factors, second derivatives, the fraction of zero activations, or how well a layer's output can be reconstructed after pruning.
Lecture 6 is about what to do when quantization costs you accuracy. First, without retraining: use finer scale granularity (per-channel, group, MX), clip outliers (EMA, calibration batches, MSE, KL), and round smarter (AdaRound). If that isn't enough, retrain: QAT keeps a full-precision copy of the weights, runs fake quantization in the forward pass, and uses the STE to pass gradients straight through. In the whitepaper table the slides cite, MobileNetV1 drops to 0.1% accuracy under per-tensor INT8 PTQ and recovers to 70.7% with per-channel QAT, against a 70.9% float baseline. The last two sections cover 1–2 bit binary and ternary networks, and HAQ, which uses reinforcement learning to assign a bit width to each layer.
Lecture 2 opens up last week's "dopey AI robot." Inputs and outputs must become numbers (tensors). Classification uses one-hot labels and softmax to turn scores into probabilities. A neural network is neurons stacked layer by layer, and training means pushing the loss down with gradient descent. The lecture ends by building a first fully connected network on MNIST in Keras and wiring it to a Gradio sketchpad. Homework 2 asks you to design your own DNN, with one hard rule: it can't have three layers. The Chang Gung satellite rubric wants a screenshot of the parameters with the best validation accuracy and encourages keeping failed attempts.
NTU Professor Yun-Nung (Vivian) Chen's Applied Deep Learning (ADL) is a deep learning course centered on NLP, with 14 lectures in four segments: neural networks and backpropagation; sequences and Transformers (word vectors, RNNs, attention, BERT); putting pretrained models to work (prompting, instruction tuning, RLHF, LoRA, RAG, decoding and evaluation); and LLM issues and frontiers (bias and safety, language agents, reasoning, tool use, multimodality). Fall 2025 slides and segmented videos are public; only HW1 has a full spec, while HW2, HW3, and the final project have only briefing videos.
The first ADL lecture has no math and sets up two ideas. First, machine learning gives a machine many input-output pairs and lets it find the function that turns one into the other; deep learning chains many simple functions into a production line, supplies data only at the two ends, and lets the machine learn what every station does. Second, to solve a problem with deep learning, write down the input X and the output Y: the form of the output decides whether the task is classification or sequence prediction, and the properties of the input decide which network fits. Along the way it answers two questions: why the breakthrough waited until after 2010 (data and GPUs), and why networks go deep instead of fat.
Training a model means answering three questions. What is the model: layers of neurons, each computing z = Wa + b followed by a non-linearity; one neuron draws only one boundary, so it cannot compute XOR, and stacking layers can. What does a good function mean: define a loss, and smaller is better. How to pick the best: gradient descent takes a step along the slope where it stands, and practice uses mini-batch SGD. With millions of parameters, gradients come from backpropagation: a forward pass stores every layer's output, a backward pass computes δ from the output layer back, and their product is each weight's gradient. The guide works each of these through with numbers.
Techniques Lecture 16 has no new model. It re-sorts the models of the first 15 lectures by three questions: where the features are hidden (kernel, aggregation, extraction, low-dimensional projection), how the error is brought down (gradients, equivalent problems, multiple steps), and how to hit the brakes when the model is too powerful (regularization, validation). This article works PCA plus AdaBoost-Stump on eight points from start to finish and checks the answer against each of the three questions; it closes by keeping the ReLU, Xavier/He initialization, momentum, RMSProp, and Adam material that the Fall 2024 on-campus course added.
Today's Deep Learning & NLP rotation covers five core concepts: why scaled dot-product attention parallelizes as a single batched matmul on GPUs, why multi-head attention splits the embedding dimension into subspaces so different heads can learn different relations, why positional encoding is information Transformers must add back once they give up recurrence, the trade-off between CNN locality, RNN recurrence, and Transformer global interaction, and how BPE, WordPiece, and SentencePiece differ as subword tokenization strategies — plus why byte-level BPE eliminates out-of-vocabulary tokens entirely. The practice question, sourced from dsprep.com's interview bank and flagged as a frequent ask at research labs like Anthropic and DeepMind, asks why Transformers chose scaled dot-product attention over additive attention — the breakdown threads efficiency, the scale factor, and the multi-head motivation into one coherent design-philosophy argument.
Lec 17 uses XOR to show that a linear model, and even a stack of linear layers, cannot learn a nonlinear boundary, while one ReLU layer can. The universal approximation theorem guarantees that a network exists but not how to find its weights or how wide it must be. Lec 18 turns the chain rule into backpropagation on a computation graph: gradients from multiple paths add up, and the cost is linear in the number of parameters, versus quadratic for finite differences. Discussion 8 has you prove a GD convergence rate and 1-D ReLU universal approximation yourself.
Lec 19 wraps up backprop, then tackles how to keep gradients flowing: all-zero initialization makes every unit learn the same thing, so use small random values (He init for ReLU), and batch norm normalizes pre-activations with mini-batch means and variances. The second half introduces CNNs: local connectivity plus weight sharing lets one feature detector scan the whole image. Lec 20 finishes pooling, receptive fields, and CNN training, then covers early stopping, dropout, and double descent, which breaks the classic bias-variance picture.
Lec 21 starts with what CNNs lack: only the top layers see the whole image. It then builds up through TF-IDF, RNN image captioning, and soft attention. Lec 22 derives self-attention from a "soft dictionary lookup": three linear layers produce Q, K, and V, the output is SoftMax(QKᵀ/√D)V, and multiple heads, an MLP, residual connections, and LayerNorm turn it into a transformer layer. Attention itself ignores order, so you need positional encodings. Discussion 10 has you compute QKV by hand and prove why the scores are divided by √D.
Lec 23 wires a transformer into a next-token predictor: tokenize, look up embeddings, stack L layers of masked attention, multiply back by the embedding table and apply softmax, and train with cross-entropy (that is, MLE). Pretraining supplies knowledge; to chat, a model also needs SFT, LoRA, RLHF, or DPO, and at inference time it leans on in-context learning, RAG, chain-of-thought, and tool calls. Lec 24 generalizes "invent a fake supervised task" to images: autoencoders, colorization, inpainting, rotation, jigsaw puzzles, clustering, and finally contrastive learning, SimCLR, and CLIP. Discussion 11 practices positional encodings, RoPE, causal masks, and the KV cache.
The last three lectures take the semester's tools to two frontiers. Lec 25 is about proteins: AlphaFold2 cracked sequence-to-structure, but the real engineering bottleneck is predicting which sequence has the function you want, and design means acting as your own model's adversary in a discrete space of size 20^L. The slides reduce conditional generation p(x|y) to three statistically correct routes, all of which come back to Bayes' rule. Lec 26 was an online guest lecture with no public materials. Lec 27 defines an agent (an LLM in a loop, using tools, deciding its next step) and argues that data is being replaced by environments: Docker + task + verifier, used for SFT, RL (RLVR, GRPO), or weight-free GEPA. For final-exam practice, use the Fall 2025 and Spring 2025 finals with solutions; the Spring 2026 final is not published.
Today's Deep Learning & NLP rotation covers five core concepts: how self-attention scores relevance with Query/Key/Value, why positional encoding is a necessary patch for self-attention's permutation-invariance, how subword tokenization (BPE/WordPiece) solves the out-of-vocabulary problem, how to answer the classic fine-tuning-vs-RAG follow-up, and how embeddings represent semantic similarity as vector distance. The practice question comes from a Google agentic AI engineer interview guide: explain how attention works and how context compression techniques affect retrieval accuracy — the breakdown threads attention's computational complexity together with the compression-vs-accuracy trade-off into one complete answer.
Today's Deep Learning & NLP rotation covers the Q/K/V computation behind self-attention and where its O(n²) complexity comes from, why positional encoding is a Transformer necessity (from sinusoidal encodings to RoPE/ALiBi), the architectural and training-objective split between BERT (encoder, bidirectional) and GPT (decoder, causal), the real difference between static word embeddings like Word2Vec/GloVe and contextual ones from BERT/GPT, and how to pick between full fine-tuning, freezing upper layers, and LoRA. The practice question asks how you'd scale a Transformer from a 2K-token to a 100K-token context window — interviewers want to see whether you can connect complexity analysis, positional extrapolation, and production cost tradeoffs in one answer.
This round of Deep Learning & NLP drills the 2026-style follow-up pattern — architecture, then engineering fallout: the four-layer decoder-only transformer (embedding + RoPE, causal self-attention, MLP, linear head), the query/key/value intuition and why causal masking matters, why decoding is memory-bandwidth bound rather than compute bound, and how token counts map directly to API cost, context window capacity, and generation latency. Today's practice question is a MAANG-level open-ended 'explain the Transformer architecture' prompt, drilling how to cover the architecture in five minutes and still land on inference-cost implications.
Deep Learning & NLP interviews don't test whether you can draw a transformer diagram — they test whether you understand what each design decision trades away. Today covers four high-frequency topics: how self-attention computes and what KV cache saves, the vocab-size trade-off in tokenization, the senior-candidate detail that 'an embedding is a vector-space contract, and matching dimensions doesn't mean compatible,' and how to choose between fine-tuning and prompting while guarding against catastrophic forgetting. The practice problem is Scale AI's 'design an embedding and classification API,' walking through version compatibility, partial failure, and multi-tenant isolation end to end.
A single hidden layer can in principle approximate any continuous function (universal approximation), but width can blow up exponentially with dimension; Barron's theorem lets error decay as 1/sqrt(n) independent of dimension for a specific function class; and depth yields exponential width savings on compositional functions — that is the real reason deep beats shallow.
A GNN is essentially 'an MLP with local message passing on a graph' — it generalizes CNN's fixed-grid neighborhood to arbitrary topology. It must satisfy permutation equivariance/invariance. In theory, a first-order GNN's expressiveness is bounded by the Weisfeiler–Lehman graph isomorphism test: some structures it can never tell apart, which is exactly the gap GIN, positional encodings, and subgraph tricks later fill.
Lecture 1 is the 6.7960 opener: deep learning took off because data + compute + algorithms matured together; the course threads from architectures (CNN/GNN/Transformer) through training, representation, generation, transfer, scaling, and LLMs; ends with a ~30-line PyTorch training loop to confirm your environment works.
Optimization is not an isolated numerical problem: view SGD spectrally, the magnitude of weight updates determines feature learning; Maximal Update Parameterization transfers LR/init across width, and the critical batch size sets the marginal return of trading compute for convergence.
A Transformer is not an architecture from nowhere: tokens discretize data, attention does soft aggregation, positional codes restore order. Seen next to MLPs/CNNs/GNNs, all of them are special cases of 'weighted aggregation over neighbors'.
Training neural nets is closer to engineering than magic: look at the data, overfit a mini-batch to prove capacity exists, then regularize back the generalization; learning rate is always the highest-leverage knob.
RNNs propagate hidden states through time; gradient products can shrink or grow. LSTM gates control cell-state retention and updates, improving long-range signal flow without guaranteeing stability. Self-attention shortens paths between positions, while dense attention still has quadratic sequence-length cost.
Representation learning compresses raw data into a 'useful' vector: autoencoders force a meaningful latent space via reconstruction, VQ discretizes it into a codebook, and self-supervision turns 'mask-and-reconstruct' into free supervision.
Similarity-based representation learning does not reconstruct input; it directly shapes latent geometry: pull same-class representations together, push different ones apart. InfoNCE turns this into 'spot the positive among negatives', and alignment / uniformity give it interpretable metrics.
Take a net to infinite width and its random-init output becomes a Gaussian process (NN–GP); its training dynamics freeze into the Neural Tangent Kernel (NTK). This theory analyzes nets and, in reverse, guides us to design the 'right inductive bias'.
Generative models learn the data distribution p(x). Density models model probability directly, energy models use an unnormalized potential + sampler, GANs let a discriminator force realistic samples, autoregressive predicts the next token step by step, and diffusion dodges tricky maximum-likelihood via 'add noise then learn to denoise'.
The core of VAE is ELBO + reparameterization: log p(x) is replaced with E_q[log p(x|z)] − KL(q(z|x)‖p(z)); the encoder outputs μ/σ and z = μ + σ⊙ε (ε ~ N(0,1)) makes sampling differentiable. Training = reconstruction + KL in tension, which gives rise to β-VAE, posterior collapse, VQ-VAE, and related fixes.
The key to conditional generation is 'feed y into the model': cGAN concatenates y into G/D; cVAE passes y to both encoder and decoder; in diffusion, Classifier Guidance uses gradients from an external classifier to push samples toward a class, while Classifier-Free Guidance trains conditional + unconditional together and linearly combines them at inference — the latter is the standard weapon behind Stable Diffusion and Imagen.
OOD failure is not a bug, it's the i.i.d. assumption breaking: covariate shift (image style changes), label shift (class proportions change), concept shift (a word's meaning changes) each need different responses; the most common cause is the model latching onto spurious correlations (using grass as a cue for cows); IRM and domain randomization try to fix this in training data structure, test-time adaptation fixes it at inference.
Transfer learning's core insight is 'features learned on big data are good general-purpose representations': freeze the backbone and train only a linear head when downstream data is tiny; full fine-tune when data is plentiful; reach for LoRA / adapter when compute is tight. SimCLR and MAE removed the need for upstream labels and pushed downstream quality another notch.
MIT 6.7960 Deep Learning (Fall 2025) publishes all 21 lecture decks as public Dropbox PDFs, and most required readings map to free textbook chapters; but the five problem sets are released only through Gradescope, and solutions plus recordings live behind Canvas login. This guide covers how the three instructors split the course, a topic map of all 21 lectures, textbook-based substitutes for lectures, and where outside self-learners realistically stop.
Stage II uses HW8 and HW11 to test whether representation, computation graphs, training, transfer, and generation actually connect, rather than treating CNNs and Transformers as diagrams to memorize.
Spring 2026 Lecture 1 focuses on neurons, perceptrons, connectionism, and the problem framing of deep learning. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 2 focuses on Boolean functions, how network depth and width affect representational capacity (including a multi-input XOR example), and the hand-off at the end to the next lecture's training problem. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 3 focuses on how the learning problem is set up (representing inputs and outputs as numbers), the perceptron learning rule and why it does not carry over to multi-layer networks, differentiable divergences, and empirical risk minimization (the loss). This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 4 focuses on the definitions of derivative and gradient, extrema and second derivatives, gradient descent as repeated steps against the gradient, and the network, input/output representations, and divergence that must be defined before training. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 5 focuses on layered-network notation, the forward pass, the chain rule and Jacobians (including vector activations), and how the backward pass computes weight and bias derivatives layer by layer. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 6 focuses on convergence of gradient descent (step-size choice on convex and quadratic functions, differences between dimensions), local minima and saddle points, decaying or adaptive learning rates, R-prop, and momentum with Nesterov accelerated gradient. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 7 focuses on incremental SGD and mini-batch updates and trend-based updates such as RMSprop and momentum (second-order methods are only in the slides, not in the recording). This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 8 focuses on the choice of divergence (loss), BatchNorm, dropout and other regularization methods, and training tricks such as early stopping (AdaGrad and Adam are not in this recording). This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 9 focuses on why we scan the input, distributing the scan over several layers, handling deformation with max pooling, and the CNN / time delay neural network structure. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 10 focuses on where the CNN comes from (Hubel and Wiesel, Fukushima's Neocognitron), the operations of convolution and pooling layers, and design choices such as filter size, stride and number of channels. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 11 focuses on a recap of CNNs and backpropagation through convolution and pooling layers. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 12 focuses on additional backpropagation rules (strides, upsampling and downsampling), invariance to transformations other than shifts, and a quick history of CNNs with the major architectures. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 13 focuses on why sequences need their own models, state and memory, Jordan/Elman-style recurrent networks, backpropagation through time, and bidirectional networks. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 14 focuses on backpropagation through time, gradient stability, and LSTM-style gated memory. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 15 focuses on the alignment problem when training sequence models (training with explicit or re-estimated alignments, and the risk of local optima); CTC is only introduced at the very end, with details left to the next lecture. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 16 focuses on blank symbols and the merging of adjacent repeated symbols, CTC's forward (alpha) probability computation, and pruned approximate decoding (beam search is described only briefly near the end). This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 17 focuses on next-word-prediction language models, conditional translation models, and beam-search decoding. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 18 focuses on sequence-to-sequence attention (queries, keys, values, and softmax-weighted averaging), a mention of multi-head attention, and a brief introduction to the Transformer. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 19 focuses on encoder/decoder structures, masks, residual paths, and architecture variants. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 20 focuses on scaled autoregressive models, pre-training and scaling laws, and post-training (including improving the model from its own sampled answers and feedback). This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 21 focuses on what neural networks actually represent, the reconstruction objective (autoencoders), and the link to PCA (dimensionality reduction). This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 22 focuses on latent variables, the ELBO, the KL term, and the reparameterization trick. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 23 focuses on forward noising, reverse denoising, score or noise prediction, and sampling. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 24 focuses on the generator, discriminator, minimax objective, and training instability. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 25 focuses on message passing, aggregation, node representations, and permutation symmetry. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 26 focuses on states, actions, rewards, returns, values, and policy learning. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 27 focuses on associative memory, energy functions, fixed points, and pattern retrieval. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
Spring 2026 Lecture 28 focuses on energy-based probability models (the Gibbs distribution and its normalizing constant Z), stochastic neurons, and how to train such probabilistic networks. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Spring 2026 publishes official slides and YouTube recordings for all 28 content lectures, plus extensive bootcamps and recitations. Its HW1–HW4 specifications, starters, and evaluation still depend on Autolab, Piazza, and Kaggle.
Winter 2026 Lecture 1 divides NLP into four eras: early exploration, symbolic systems, statistical machine learning, and deep/self-supervised learning. The point is not the dates but how each era redefined the language problem.
Lecture 2 of the 2026 course addresses data where order changes meaning—text, audio, and time series—and connects directly to music generation in Lab 1.
Lecture 3 of the 2026 course moves from image tensors, convolution, and pooling to recognition systems, preparing for MNIST and face detection in Lab 2.
Lecture 4 of the 2026 course separates generative from discriminative tasks, organizes VAE, GAN, and diffusion objectives, and leads into Lab 2’s DB-VAE.
Lecture 5 of the 2026 course connects agent, environment, state, action, reward, and policy into an interaction loop, introducing credit assignment and exploration.
Lecture 6 of the 2026 course places deep learning in emerging applications and real constraints, emphasizing data, outputs, evaluation, and failure conditions.
Lecture 7 of the 2026 course starts from Asimov’s literary laws and examines modern safety protocols through traces, test data, and continuous evaluation.
Lecture 8 of the 2026 course uses the scientific-discovery loop to show how simulators, AI emulators, and experiments cooperate instead of reducing science to generic prediction.
Lecture 9 of the 2026 course starts with GPU memory pressure and moves through checkpointing, offloading, ZeRO, FSDP, and multiple forms of parallelism.
In the 2026 lab, part 1 classifies MNIST with dense and convolutional networks; Part 2 learns a facial latent distribution with a DB-VAE and changes training sampling.
In the 2026 lab, students build chat templates and generation with LFM2-1.2B, adapt style through LoRA, and combine OpenRouter with Opik for a judge workflow.
Lectures 19–25 connect rational decisions and VPI to machine learning, while Project 5 uses PyTorch for regression, classification, CNNs, attention, and an optional character-GPT.
MIT 6.S191's 2026 edition publishes nine lecture videos, slides, three software labs, and solutions, making it an A3 self-study course. The supplied path still depends on Google/Colab, Comet, and OpenRouter for Lab 3, while unaffiliated learners do not receive MIT credit, project feedback, or API credits.
CS229 is the Stanford course that works through the mathematical derivations of machine learning. The Spring 2026 recordings make 17 lectures public: the first ten cover linear models, neural networks, and unsupervised learning, and the last seven go on to diffusion models, LLMs, and training reasoning models with reinforcement learning. This series has one post per lecture, and this post is the map: what each lecture actually teaches, where the video titles don't match, and where to pick up what the lecture notes cover but the lectures skipped.
Deep learning interviews don't ask you to derive backpropagation — they test whether you can explain the design intuition behind architectures. High-frequency topics: CNN's locality and translation invariance, why the evolution from RNN to Transformer was necessary, self-attention computation and complexity, BatchNorm vs LayerNorm use cases, and common training tricks (learning rate scheduling, gradient clipping, mixed precision).
CS230's first lecture is a course overview, but Andrew Ng spends most of it on three things: why scaling works, when prompting stops being enough, and why he thinks 'don't learn to code' is one of the worst pieces of career advice ever given.