Skip to content

MIT 6.7960 L09: Hacker's Guide to Deep Learning — Practical Know-How to Make Nets Actually Obey

Aug 30, 2026 1 min
Home › tech
TL;DR Training neural nets is closer to engineering than magic: look at the data, overfit a mini-batch to prove capacity exists, then regularize back the generalization; learning rate is always the highest-leverage knob.
#mit-67960#deep-learning#training-tricks#debugging#fall-2024
Table of Contents
  1. What gap this lecture fills
  2. 1. Data first
  3. 2. Overfit a mini-batch first
  4. 3. Learning rate: the highest-leverage knob
  5. 4. Regularization is "putting generalization back"
  6. 5. A reproducible recipe
  7. 6. Monitoring and a debugging checklist
  8. Why this matters in practice
  9. References
Series: MIT 6.7960 導讀 (Fall 2024 OCW) (11 / 16)
MIT 6.7960 L08: Transformers — Tokens, Attention, Positional Codes, and How They Relate to MLPs/CNNs/GNNs MIT 6.7960 L10: Memory and Sequence Modeling — RNNs, LSTMs, and Vanishing/Exploding Gradients
Series: Global AI/CS Course Map (17 / 29)
MIT 6.7960 L08: Transformers — Tokens, Attention, Positional Codes, and How They Relate to MLPs/CNNs/GNNs MIT 6.7960 L10: Memory and Sequence Modeling — RNNs, LSTMs, and Vanishing/Exploding Gradients

🌏 中文版

Source version: based on MIT 6.7960 Fall 2024 OCW. Videos, slides, and assignments are public at MIT OCW. This lecture is taught by Phillip Isola; references include Recipes for Training Neural Networks and Google's Rules of ML.

What gap this lecture fills

Earlier lectures covered architectures, optimization, and regularization — all "correct knowledge", but none told you what to do when the loss refuses to move for three days. Lecture 9 is that "survival manual for practice".

The core mindset: training neural networks is engineering, not alchemy. It can be systematized, debugged, and reduced to a repeatable procedure.

1. Data first

The vast majority of "model doesn't work" problems trace back to data:

  • Look at the data yourself: sample a few dozen examples, confirm labels are sane, no obvious mislabels, input ranges are normal.
  • Check train/val distribution: if the validation set diverges from training, no model will generalize.
  • Build the simplest baseline: a trivial rule (e.g., "always predict the majority class") gives you the accuracy floor to beat.

2. Overfit a mini-batch first

The most underrated step. Before any real training, take a tiny number of samples (even one batch) and tune the model until it memorizes them (train loss → 0).

  • If you can't: capacity too small, LR wrong, or a bug in data/forward pass. Fix this before talking about generalization.
  • If you can: your pipeline is sound; what remains is just balancing capacity against regularization.

This single step eliminates ~80% of low-level errors (dimension mismatches, dead activations, vanishing gradients) within five minutes.

3. Learning rate: the highest-leverage knob

If you can tune only one hyperparameter, tune the learning rate. Practical guidance:

  • Sweep a range first (e.g., 1e-1, 1e-2, 1e-3, 1e-4) to see which makes loss drop stably.
  • Use warmup + decay: a small LR early avoids destroying random init, ramp up, then decay late.
  • When batch grows, LR usually grows too (linear scaling rule: batch ×k → LR ≈ ×k, up to the critical batch size).

4. Regularization is "putting generalization back"

After overfitting the mini-batch proves capacity exists, regularize the generalization back:

  • Dropout: randomly disable neurons during training, forcing redundant representations.
  • Weight decay (L2): penalize large weights, making the solution smoother.
  • Data augmentation: synthesize data without changing label semantics — the cheapest regularization.
  • Early stopping: stop when validation loss starts rising; effectively regularizing on "training steps".

The always-valid diagnostic is the train/val loss gap: large gap = overfitting (add regularization); high train loss itself = underfitting (add capacity or train longer).

5. A reproducible recipe

Turn the above into an order you follow every time:

  1. Look at data, build a baseline.
  2. Overfit a mini-batch, confirm the pipeline works.
  3. Run on full data at medium capacity, tuning only the learning rate.
  4. Inspect the train/val gap, decide between more regularization or more capacity.
  5. Only then do "fine-tuning": activation choice, normalization style, architectural details.

Never start with the fanciest architecture plus a pile of tricks — that leaves you unable to tell which change actually helped.

6. Monitoring and a debugging checklist

When loss won't drop, check in this order:

  • Is the LR too large (loss oscillates/diverges) or too small (flatlined)?
  • Are inputs properly normalized (mean 0, variance 1)?
  • Did a whole layer of activations die (all zero after ReLU)?
  • Are gradients vanishing/exploding (check gradient norm)?
  • Did the data pipeline corrupt labels or break shuffling?

Why this matters in practice

The value of this lecture is not any single trick but establishing a controllable experiment flow. With a repeatable recipe, every failed experiment becomes diagnosable instead of "try a different random seed".

References

Related

MIT 6.7960 L06: Modern CNN Architectures — ResNet, EfficientNet, ConvNeXt 1 min MIT 6.7960 L07: Scaling Rules for Optimization — Spectral View, Feature Learning, Hyperparameter Transfer 1 min MIT 6.7960 L08: Transformers — Tokens, Attention, Positional Codes, and How They Relate to MLPs/CNNs/GNNs 1 min