CMU 10-301 HW7: Move from Basic Neural Networks to Deep Learning

TL;DR HW7 builds on HW5 backpropagation to address deep-model architecture and training failures, emphasizing diagnosis over merely adding layers.

#cmu #machine-learning #deep-learning #neural-network

Table of Contents

Establish a diagnostic order
First executable action and completion
References

Series: Reading CMU 10-301 Machine Learning (7 / 10)

CMU 10-301 HW6: Learning Theory, MLE/MAP, and Fairness Metrics CMU 10-301 HW8: From MDPs to Reinforcement-Learning Updates

🌏 中文版

The official handout is titled Homework 7: Deep Learning and contains written plus programming work. Written sections cover CNNs, RNNs, Transformers and AutoDiff, and empirical questions. Programming implements an RNN cell, self-attention, hybrid language model, validation, and generation on a TinyStories subset. The ZIP supplies rnn.py, public tests/test data, a tokenizer, tiny train/validation stories, loss/metric references, a Colab notebook, and an environment file.

Establish a diagnostic order

Fix a tiny dataset and random seed. Check shapes, initial loss, one-step gradients, and tiny-set overfitting before architecture comparisons. If training loss does not move, inspect activations, normalization, and learning rate. Discuss regularization only when training succeeds and validation fails.

First executable action and completion

Start with the bundle's environment file:

conda env create -f environment.yml

Then run test_runner.py and validate/generate on tiny stories. Completion means passing public tests for the RNN cell, attention, hybrid LM, training, validation, and generation, and reproducing tiny loss/metric references within their expected tolerance. Public tests do not replace Gradescope hidden tests.

CMU 10-301 HW7: Move from Basic Neural Networks to Deep Learning

Establish a diagnostic order

First executable action and completion

References

Related