MIT 6.7960 L03: Optimization Overview — SGD, Adam, LR Schedules & Scaling Rules
From SGD to Adam: pick the right optimizer and scale LR with batch size using scaling rules
#deep-learning
72 postsMIT 6.7960 L04: Regularization in Practice — Weight Decay, Dropout, Batch Norm & Label Smoothing
Regularization isn't just anti-overfitting — mechanisms & combo strategies for WD, Dropout, BN, Label Smoothing
MIT 6.7960 PS1 Walkthrough: From NumPy MLP to PyTorch Autograd Backprop
Hand-write NumPy MLP + backprop → verify with PyTorch Autograd, fully reproducing OCW HW1 core concepts
MIT 6.7960 L05: CNN Architectures — From Convolution Kernels to Translation Equivariance
Lec 4 core: why CNN is the natural choice for grid data — convolution, translation equivariance, pooling, and classic architectures in one go
MIT 6.7960 L06: Modern CNN Architectures — ResNet, EfficientNet, ConvNeXt
ResNet's skip connections solve degradation, enabling 100+ layer nets; EfficientNet compound scales depth/width/resolution; ConvNeXt absorbs Transformer design to reclaim CV crown.
MIT 6.7960 L07: Scaling Rules for Optimization — Spectral View, Feature Learning, Hyperparameter Transfer
Optimization is not an isolated numerical problem: view SGD spectrally, the magnitude of weight updates determines feature learning; Maximal Update Parameterization transfers LR/init across width, and the critical batch size sets the marginal return of trading compute for convergence.
MIT 6.7960 L08: Transformers — Tokens, Attention, Positional Codes, and How They Relate to MLPs/CNNs/GNNs
A Transformer is not an architecture from nowhere: tokens discretize data, attention does soft aggregation, positional codes restore order. Seen next to MLPs/CNNs/GNNs, all of them are special cases of 'weighted aggregation over neighbors'.
MIT 6.7960 L09: Hacker's Guide to Deep Learning — Practical Know-How to Make Nets Actually Obey
Training neural nets is closer to engineering than magic: look at the data, overfit a mini-batch to prove capacity exists, then regularize back the generalization; learning rate is always the highest-leverage knob.
MIT 6.7960 L10: Memory and Sequence Modeling — RNNs, LSTMs, and Vanishing/Exploding Gradients
An RNN compresses the past into a hidden state, but recurrence makes gradients multiply over time — they either vanish or explode; LSTM decouples 'memory' from 'update' via input/forget/output gates so long-range information flows stably. Attention later replaced it because it reaches any history in O(1).
MIT 6.7960 L11: Representation Learning (Reconstruction-Based) — Autoencoders, VQ, Self-Supervision
Representation learning compresses raw data into a 'useful' vector: autoencoders force a meaningful latent space via reconstruction, VQ discretizes it into a codebook, and self-supervision turns 'mask-and-reconstruct' into free supervision.
MIT 6.7960 L12: Representation Learning (Similarity-Based) — Metric Learning, Contrastive, InfoNCE
Similarity-based representation learning does not reconstruct input; it directly shapes latent geometry: pull same-class representations together, push different ones apart. InfoNCE turns this into 'spot the positive among negatives', and alignment / uniformity give it interpretable metrics.
MIT 6.7960 L13: Theory of Representation — Inductive Biases, Gaussian Processes, and the NN–GP Correspondence
Take a net to infinite width and its random-init output becomes a Gaussian process (NN–GP); its training dynamics freeze into the Neural Tangent Kernel (NTK). This theory analyzes nets and, in reverse, guides us to design the 'right inductive bias'.
MIT 6.7960 L14: Generative Models Basics — Density/Energy Models, GANs, Autoregressive, Diffusion
Generative models learn the data distribution p(x). Density models model probability directly, energy models use an unnormalized potential + sampler, GANs let a discriminator force realistic samples, autoregressive predicts the next token step by step, and diffusion dodges tricky maximum-likelihood via 'add noise then learn to denoise'.
MIT 6.7960 Approximation Theory — Universal Approximation, Barron's Theorem, and Why Depth Matters
A single hidden layer can in principle approximate any continuous function (universal approximation), but width can blow up exponentially with dimension; Barron's theorem lets error decay as 1/sqrt(n) independent of dimension for a specific function class; and depth yields exponential width savings on compositional functions — that is the real reason deep beats shallow.
MIT 6.7960 Graph Neural Networks (GNN) — Message Passing, Permutation Equivariance, and the Expressiveness Ceiling
A GNN is essentially 'an MLP with local message passing on a graph' — it generalizes CNN's fixed-grid neighborhood to arbitrary topology. It must satisfy permutation equivariance/invariance. In theory, a first-order GNN's expressiveness is bounded by the Weisfeiler–Lehman graph isomorphism test: some structures it can never tell apart, which is exactly the gap GIN, positional encodings, and subgraph tricks later fill.
MIT 6.7960 L15: Variational Autoencoders (VAE) — ELBO, Reparameterization Trick, and Latent Representations
The core of VAE is ELBO + reparameterization: log p(x) is replaced with E_q[log p(x|z)] − KL(q(z|x)‖p(z)); the encoder outputs μ/σ and z = μ + σ⊙ε (ε ~ N(0,1)) makes sampling differentiable. Training = reconstruction + KL in tension, which gives rise to β-VAE, posterior collapse, VQ-VAE, and related fixes.
MIT 6.7960 L16: Conditional Generative Models — cGAN, cVAE, and Classifier-Free Guidance
The key to conditional generation is 'feed y into the model': cGAN concatenates y into G/D; cVAE passes y to both encoder and decoder; in diffusion, Classifier Guidance uses gradients from an external classifier to push samples toward a class, while Classifier-Free Guidance trains conditional + unconditional together and linearly combines them at inference — the latter is the standard weapon behind Stable Diffusion and Imagen.
MIT 6.7960 L01: Course Introduction — A Map of Deep Learning, Why Depth Works, and Your First Training Loop
Lecture 1 is the 6.7960 opener: deep learning took off because data + compute + algorithms matured together; the course threads from architectures (CNN/GNN/Transformer) through training, representation, generation, transfer, scaling, and LLMs; ends with a ~30-line PyTorch training loop to confirm your environment works.
MIT 6.7960 L17: Out-of-Distribution Generalization — Distribution Shift, Spurious Correlations, and Three Practical Remedies
OOD failure is not a bug, it's the i.i.d. assumption breaking: covariate shift (image style changes), label shift (class proportions change), concept shift (a word's meaning changes) each need different responses; the most common cause is the model latching onto spurious correlations (using grass as a cue for cows); IRM and domain randomization try to fix this in training data structure, test-time adaptation fixes it at inference.
MIT 6.7960 L18: Transfer Learning — Pretraining, Feature Extraction, and Fine-Tuning Strategies
Transfer learning's core insight is 'features learned on big data are good general-purpose representations': freeze the backbone and train only a linear head when downstream data is tiny; full fine-tune when data is plentiful; reach for LoRA / adapter when compute is tight. SimCLR and MAE removed the need for upstream labels and pushed downstream quality another notch.
Reading MIT 6.7960: One Course, Two Official Editions — Complete the OCW 2024 Package, Read the 2025 Decks for What's New
MIT 6.7960 Deep Learning (Fall 2025) publishes all 21 lecture decks as public Dropbox PDFs, and most required readings map to free textbook chapters; but the five problem sets are released only through Gradescope, and solutions plus recordings live behind Canvas login. This guide covers how the three instructors split the course, a topic map of all 21 lectures, textbook-based substitutes for lectures, and where outside self-learners realistically stop.
CMU 07-280 Stage Review II: Building AlexNet and GPT-2 as Working Systems
Stage II uses HW8 and HW11 to test whether representation, computation graphs, training, transfer, and generation actually connect, rather than treating CNNs and Transformers as diagrams to memorize.
CMU 11-785 Lecture 1: Introduction
Spring 2026 Lecture 1 focuses on neurons, perceptrons, connectionism, and the problem framing of deep learning. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 2: Neural Nets as Universal Approximators
Spring 2026 Lecture 22 focuses on latent variables, the ELBO, the KL term, and the reparameterization trick. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 3: Training I: Learning and Empirical Risk Minimization
Spring 2026 Lecture 3 focuses on data distributions, hypotheses, losses, empirical risk, and their roles in generalization. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 4: Training II: Gradient Descent
Spring 2026 Lecture 4 focuses on gradients, learning rates, parameter updates, and the training of a linear neuron. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 5: Training III: Backpropagation
Spring 2026 Lecture 5 focuses on computational graphs, the chain rule, local derivatives, and gradient reuse. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 6: Training IV: Convergence, Loss Surfaces, and Momentum
Spring 2026 Lecture 6 focuses on non-convex loss surfaces, curvature, saddle points, and momentum's accumulated direction. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 7: Training V: SGD and Second-order Methods
Spring 2026 Lecture 7 focuses on the tradeoffs among full-batch, mini-batch, stochastic gradients, and second-order information. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 8: Training VI: Optimizers and Regularization
Spring 2026 Lecture 8 focuses on AdaGrad, Adam, regularization, BatchNorm, Dropout, and loss selection. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 9: CNNs I
Spring 2026 Lecture 9 focuses on local connectivity, weight sharing, convolution kernels, and feature maps. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 10: CNNs II
Spring 2026 Lecture 10 focuses on stride, padding, receptive fields, and multi-channel convolution. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 11: CNNs III
Spring 2026 Lecture 11 focuses on stacked convolutional architectures, feature hierarchies, and design tradeoffs. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 12: CNNs IV
Spring 2026 Lecture 12 focuses on CNN training, architecture selection, and the end-to-end assembly of a vision model. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 13: RNNs I
Spring 2026 Lecture 13 focuses on sequence state, temporal unrolling, parameter sharing, and recurrent computation. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 14: RNNs II
Spring 2026 Lecture 14 focuses on backpropagation through time, gradient stability, and LSTM-style gated memory. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 15: Seq2Seq and Connectionist Temporal Classification
Spring 2026 Lecture 15 focuses on variable-length input/output, unknown alignment, and the CTC objective. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 16: CTC Blanks and Beam Search
Spring 2026 Lecture 16 focuses on blanks, collapse rules, prefix probabilities, and approximate decoding. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 17: Language Models and Translation
Spring 2026 Lecture 17 focuses on autoregressive factorization, conditional language models, and translation decoding. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 18: Attention and Transformers
Spring 2026 Lecture 18 focuses on queries, keys, values, scaled dot-product attention, and the Transformer block. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 19: Transformers and Newer Architectures
Spring 2026 Lecture 19 focuses on encoder/decoder structures, masks, residual paths, and architecture variants. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 20: Large Language Models
Spring 2026 Lecture 20 focuses on scaled autoregressive models, training stages, inference, and capability boundaries. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 21: Representations and Autoencoders
Spring 2026 Lecture 21 focuses on bottleneck representations, reconstruction objectives, dimensionality reduction, and representation quality. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 22: Variational Autoencoders
Spring 2026 Lecture 22 focuses on latent variables, the ELBO, the KL term, and the reparameterization trick. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 23: Diffusion Models
Spring 2026 Lecture 23 focuses on forward noising, reverse denoising, score or noise prediction, and sampling. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 24: Generative Adversarial Networks
Spring 2026 Lecture 24 focuses on the generator, discriminator, minimax objective, and training instability. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 25: Graph Neural Networks
Spring 2026 Lecture 25 focuses on message passing, aggregation, node representations, and permutation symmetry. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 26: Reinforcement Learning
Spring 2026 Lecture 26 focuses on states, actions, rewards, returns, values, and policy learning. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 27: Hopfield Networks
Spring 2026 Lecture 27 focuses on associative memory, energy functions, fixed points, and pattern retrieval. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
CMU 11-785 Lecture 28: Boltzmann Machines
Spring 2026 Lecture 28 focuses on energy-based probability models, stochastic units, the partition function, and learning difficulty. This guide follows the official slides and recording and adds a small self-check that does not depend on the enrolled-course grader.
A Complete Guide to CMU 11-785: 28 Public Lectures, but an Incomplete Assignment Chain
CMU 11-785 Spring 2026 publishes official slides and YouTube recordings for all 28 content lectures, plus extensive bootcamps and recitations. Its HW1–HW4 specifications, starters, and evaluation still depend on Autolab, Piazza, and Kaggle.
CS224N Lecture 1: Four Paradigm Shifts in NLP
Winter 2026 Lecture 1 divides NLP into four eras: early exploration, symbolic systems, statistical machine learning, and deep/self-supervised learning. The point is not the dates but how each era redefined the language problem.
MIT 6.S191 Lecture 1: The Minimal Structure of Deep Learning
Lecture 1 of the 2026 course builds the vocabulary shared by the rest of the course: perceptrons, forward propagation, loss, and gradient descent.
MIT 6.S191 Lecture 2: Sequence Modeling: From RNNs to Attention
Lecture 2 of the 2026 course addresses data where order changes meaning—text, audio, and time series—and connects directly to music generation in Lab 1.
MIT 6.S191 Lecture 3: Computer Vision: How Convolution Preserves Spatial Structure
Lecture 3 of the 2026 course moves from image tensors, convolution, and pooling to recognition systems, preparing for MNIST and face detection in Lab 2.
MIT 6.S191 Lecture 4: Generative Modeling: From Latent Spaces to Diffusion
Lecture 4 of the 2026 course separates generative from discriminative tasks, organizes VAE, GAN, and diffusion objectives, and leads into Lab 2’s DB-VAE.
MIT 6.S191 Lecture 5: Reinforcement Learning: Learning from Return Instead of Labels
Lecture 5 of the 2026 course connects agent, environment, state, action, reward, and policy into an interaction loop, introducing credit assignment and exploration.
MIT 6.S191 Lecture 6: New Frontiers: Choosing the Problem Beyond the Model
Lecture 6 of the 2026 course places deep learning in emerging applications and real constraints, emphasizing data, outputs, evaluation, and failure conditions.
MIT 6.S191 Lecture 7: The Three Laws of AI: Safety Through Observability and Evaluation
Lecture 7 of the 2026 course starts from Asimov’s literary laws and examines modern safety protocols through traces, test data, and continuous evaluation.
MIT 6.S191 Lecture 8: AI for Science: Putting Domain Structure into Learning
Lecture 8 of the 2026 course uses the scientific-discovery loop to show how simulators, AI emulators, and experiments cooperate instead of reducing science to generic prediction.
MIT 6.S191 Lecture 9: Massively Parallel Training: Memory and Communication Set the Boundary
Lecture 9 of the 2026 course starts with GPU memory pressure and moves through checkpointing, offloading, ZeRO, FSDP, and multiple forms of parallelism.
MIT 6.S191 Lab 1: Generate Music with PyTorch and an LSTM
In the 2026 lab, students cover tensors, autograd, and modules before turning ABC notation into character sequences for LSTM music generation.
MIT 6.S191 Lab 2: From MNIST to Facial Debiasing with a DB-VAE
In the 2026 lab, part 1 classifies MNIST with dense and convolutional networks; Part 2 learns a facial latent distribution with a DB-VAE and changes training sampling.
MIT 6.S191 Lab 3: LoRA Fine-Tuning and LLM-as-a-Judge Evaluation
In the 2026 lab, students build chat templates and generation with LFM2-1.2B, adapt style through LoRA, and combine OpenRouter with Opik for a judge workflow.
Deep Learning: Modules, Backpropagation, and Vectorization
Chapter 7 decomposes neural networks into composable modules and uses backpropagation and vectorization to explain how deep models can be trained efficiently.
CS188 Decisions and Machine Learning: From VPI and Naive Bayes to Attention
Lectures 19–25 connect rational decisions and VPI to machine learning, while Project 5 uses PyTorch for regression, classification, CNNs, attention, and an optional character-GPT.
CMU 10-301 HW7: Move from Basic Neural Networks to Deep Learning
HW7 builds on HW5 backpropagation to address deep-model architecture and training failures, emphasizing diagnosis over merely adding layers.
Stanford CS109 Lecture 22 | Deep Learning: Derive backpropagation with the chain rule
A neural network stacks logistic units; a forward pass computes probabilities, while backpropagation reuses output error to obtain every gradient.
MIT 6.S191 Guide: Nine Lectures and Three Labs Are Public, but the Full Path Still Uses Three External Services
MIT 6.S191's 2026 edition publishes nine lecture videos, slides, three software labs, and solutions, making it an A3 self-study course. The supplied path still depends on Google/Colab, Comet, and OpenRouter for Lab 3, while unaffiliated learners do not receive MIT credit, project feedback, or API credits.
Stanford CS229: Notes Rewritten Every Year, Public Problem Sets Frozen at 2020, and an Official Self-Test From 2008
The three things you need to self-study CS229 run on three different clocks. The lecture notes are 278 pages and were recompiled in August 2026. The newest problem sets you can download are from summer 2020. The self-assessment Stanford Online tells you to attempt before enrolling is a PDF created in 2008. Seventeen lectures from spring 2026 are public, and the last three are mislabeled.
Deep Learning Interview Guide: Core Intuitions from CNN to Transformer
Deep learning interviews don't ask you to derive backpropagation — they test whether you can explain the design intuition behind architectures. High-frequency topics: CNN's locality and translation invariance, why the evolution from RNN to Transformer was necessary, self-attention computation and complexity, BatchNorm vs LayerNorm use cases, and common training tricks (learning rate scheduling, gradient clipping, mixed precision).
Introduction to Deep Learning: The Two Moments Prompting Stops Being Enough
CS230's first lecture is a course overview, but Andrew Ng spends most of it on three things: why scaling works, when prompting stops being enough, and why he thinks 'don't learn to code' is one of the worst pieces of career advice ever given.