Testing minGPT Changes: Shapes, Masks, and Positions | CMU 10-423 Assignment 1
Assignment 1 brings sequence modeling and modern attention into code.
#pytorch
31 postsTesting DDPM: Matching Timestep Coefficients to a Batch | CMU 10-423 Assignment 2
Copying a noising equation is not enough to implement diffusion correctly.
RNN Language Models: Sequence Probabilities and Gradient Paths | CMU 10-423 Lecture 1
A sequence such as “I like cats” is not a bag of independent words.
Distributed Training and Data Parallelism, and Where Gradient Sync Costs Come From | CMU 11-868 Lectures 14–15
Lectures 14 and 15 of 11-868 go from the parameter server to PyTorch DDP. They use NCCL's five collectives (Broadcast, Reduce, AllReduce, ReduceScatter, AllGather) as building blocks, show why a ring makes broadcast time nearly independent of GPU count, and split AllReduce into ReduceScatter plus AllGather. The second lecture takes apart DDP's two key designs: bucketing gradients (25 MB by default) and starting synchronization before the backward pass finishes. The official syllabus lists no public recording link; this guide works from slide page numbers and the VLDB 2020 paper.
How a Deep Learning Framework Computes Gradients from a Computation Graph | CMU 11-868 Lecture 5
L05 follows a small sentiment classification network throughout. It expresses computation as a graph, evaluates it in topological order, sends gradients back with the chain rule and vector-Jacobian products, and then takes apart TensorFlow v1's placeholder, variable, operation, and session. One slide is labeled "important for HW2".
Writing Data Parallelism and Pipeline Parallelism Yourself on Two GPUs | CMU 11-868 Assignment 5
CMU 11-868's fifth assignment switches to PyTorch and Hugging Face GPT-2. Using only torch.distributed and torch.multiprocessing, you write data parallelism (partition the data, set up a process group, average gradients; 50 points), then a GPipe-style pipeline (split the model, generate a clock schedule, run micro-batches on worker threads; 50 points). Both parts need benchmarks and plots on at least two GPUs: data parallelism must reach at least 1.5x speedup on 2 GPUs, and the pipeline must beat plain model parallelism. The Spring 2026 deadline was 3/25.
BatchNorm, Dropout, CNNs, PyTorch, and RNN Captioning | Stanford CS231N Assignment 2
Assignment 2 of CS231N Spring 2026 is worth 18% of the grade. Across five notebooks you hand-write BatchNorm/LayerNorm, dropout, and the forward and backward passes for convolution and pooling, then learn PyTorch at three levels of abstraction, and finish with RNN image captioning on COCO in PyTorch. Q4 is the turning point: through Q3 you derive every gradient yourself, and from Q5 on autograd takes over while a numerical gradient check confirms it. The official slides warn that this is the longest of the three assignments.
How Do You Measure a Model's Size? Parameters, Activations, MACs, and Latency | MIT 6.5940 Lectures 1–2 / Lab 0
The first two lectures of 6.5940 show that the problem exists, then hand you the rulers. L1 plots model parameter counts growing much faster than GPU memory, and contrasts 80GB on a cloud GPU with 320kB on a microcontroller. L2 splits efficiency metrics into memory metrics (#parameters, model size, peak activations) and compute metrics (MAC, FLOP, OP). AlexNet has 61M parameters and 724M MACs, and on a microcontroller the thing that runs out first is usually activation memory, not parameters. Lab 0 introduces a VGG variant on CIFAR-10 (9.2M parameters, 606M MACs) that later labs build on.
Course Summary and Quantum Machine Learning: Pruning, NAS, and On-Device Training on Quantum Circuits | MIT 6.5940 Lectures 22–23
The last two lectures of MIT 6.5940 Fall 2024 come in two parts. The opening of Lecture 22 is a 13-page Course-Summary.pdf (only a short stretch at the start of the recording) that redraws the course as three blocks (inference, training, application-specific) on System and Algorithm axes, then lays out the 7-item final project rubric. Most of the rest of Lecture 22 is Quantum ML Part I, which has a recording but no slides. Lecture 23 (Hanrui Wang, 99 slides) covers parameterized quantum circuits (PQCs): data encoding, parameter-shift gradients, probabilistic gradient pruning under noise (QOC), the TorchQuantum library, and QuantumNAS, which searches with a SuperCircuit and then prunes gates. It reads like a replay of the course's supernet and magnitude pruning on quantum circuits. Fall 2026 has replaced both lectures with a guest lecture.
Nine Questions on Fine-Grained and Channel Pruning | MIT 6.5940 Lab 1
MIT 6.5940 Fall 2024 Lab 1 is one Colab notebook with 9 questions worth 100 points. Questions 1–5 apply magnitude-based fine-grained pruning and a sensitivity scan to a VGG on CIFAR-10, and require a model at 25% of its original size with over 92.5% accuracy after fine-tuning. Questions 6–8 cover channel pruning, Frobenius-norm channel ranking, and measured speedup; Question 9 compares the two. Fall 2026 has no pruning lab.
Implementing K-means and Linear Quantization, Down to an Integer-Only VGG | MIT 6.5940 Lab 2
Lab 2 is a Colab notebook with 10 questions worth 100 points, built around a VGG pretrained on CIFAR-10. The first 3 questions cover K-means quantization: write the quantizer, work out how many clusters n bits gives you, write the centroid update, then compare accuracy at 8, 4, and 2 bits before and after fine-tuning. The other 7 cover linear quantization: write q = round(r/S) + Z, derive the scale and zero-point formulas, do per-channel weight quantization and bias quantization, write integer versions of the fully connected and convolution layers, and finally convert the whole model to INT8 for inference. This post lays out the questions, points, setup, and limits for outside learners. No solutions.
Writing Flow Matching and Score Matching by Hand | MIT 6.S184 Lab 2
Lab 2 turns §3–4 of the notes into PyTorch. You implement the Gaussian conditional path, its conditional vector field, and its conditional score. Then two nearly identical trainers do flow matching and score matching, Proposition 1 converts the learned vector field into a score, and finally a linear path makes a ring distribution flow into a checkerboard. Everything runs on 2D toy data. The README records a diffusion-coefficient bug fix dated 1/11/26; when I checked on 2026-09-30, the fix appeared only in the solutions notebook, not the student version, so patch it yourself before you start.
NTHU NLP Guide 11: GPT-2 vs. T5 for Chinese Summarization — Left Padding, −100, and ROUGE | NTHU Hung-Yu Kao Natural Language Processing Topic Guide
A guide to the GPT-2 / T5 tutorial in NTHU Prof. Hung-Yu Kao's NLP course (Fall 2025). One task, LCSTS Chinese summarization, is solved twice. Decoder-only GPT-2 is written in native PyTorch: you join article and summary into one sequence, switch to left padding, and set padding labels to −100. Encoder-decoder mT5 uses Seq2SeqTrainer: no left padding needed, and DataCollatorForSeq2Seq handles the −100 for you. Both segment with jieba and score word-level ROUGE.
NTHU NLP Guide 9: One BERT That Scores and Classifies at Once — The Hugging Face Tutorial and HW3 Multi-Output Learning | NTHU Hung-Yu Kao Natural Language Processing Topic Guide
A guide to the Hugging Face BERT tutorial and HW3 in NTHU Prof. Hung-Yu Kao's NLP course (Fall 2025). The tutorial walks through binary IMDb sentiment classification: AutoTokenizer, the input_ids / token_type_ids / attention_mask fields, AutoModelForSequenceClassification, and Trainer. HW3 applies the same tools to SemEval 2014 Task 1: one bert-base-uncased with two heads, one regressing a 1–5 relatedness score and one classifying entailment into three classes. You add the two losses and write the training loop yourself, because Trainer is not allowed.
Arithmetic as a Language — PyTorch TA Session, a Two-Layer LSTM, and Teacher Forcing | NTHU Hung-Yu Kao Natural Language Processing Assignment 2
HW2 treats expressions like "14*(43+20)=882" as character sequences and asks a two-layer LSTM to generate the answer one character at a time after it sees "=". The training set has 2,369,250 rows and the eval set 263,250, with every number in 0–49. Six TODOs run from building a vocabulary and batching with loss only after "=" to a generator, teacher-forced training, and exact-match evaluation. The W4 PyTorch TA session is the toolbox for it.
From PyTorch and Hugging Face to LoRA, Quantization, and vLLM Deployment | NTU Yun-Nung Chen Applied Deep Learning Tutorial
The ADL Fall 2025 course page schedules seven TA recitations across ten videos: Colab and PyTorch, ipdb debugging → the life of an NLP project → data formats for four NLP task types → LoRA and QLoRA → scaling laws, the LLaMA architecture, and MoE → quantization and evaluation → deployment with vLLM. The videos were recorded by the TA in 2023 and 2024 and reused in Fall 2025, so the Homework 1 and Homework 3 they mention are those years' assignments. The course page's five slide links all return 404; files with the same names open under the Fall 2024 path, and Deployment has a video only. This guide follows what the TA actually demonstrates and warns about in each session, with examples you can work by hand.
Drawing Pokémon with Next-Token Prediction on a Decoder-Only Transformer | NTU Hung-yi Lee Machine Learning Assignment 4
HW4 moves next-token prediction from text to images unchanged: 792 Pokémon sprites at 20×20, each pixel one of 167 color tokens, so one image is a 400-token sequence. Training is next-token prediction; at test time you get the first 60% of an image and the model draws the rest. Grading checks FID and a Pokémon Detection Rate (PDR) together, and the three baseline hints go from "run the sample code" to "tune hyperparameters" to "switch to Llama or Mistral". The spec, Colab, Kaggle notebook and dataset are public, but JudgeBoi returned 502 on 2026-09-30, so outside readers cannot get official FID or PDR scores.
ResNet/Transformer Paper Questions and Implementing CNN, ResNet, Transformer, DNABERT, and ConvNeXt | Berkeley CS189 Assignment 4
HW4 has three pieces. The written part has you read ResNet and Attention Is All You Need in the order problem → existing work → proposal → method → contribution. The 4.1 notebook builds a CNN and ResNet-18 in PyTorch, then assembles an encoder-decoder transformer step by step from softmax, trains it on TinyStories, and generates stories. The 4.2 notebook cuts DNA into 6-mers for a pretrained DNABERT to classify species, and turns audio into spectrograms for ConvNeXt, comparing training from scratch, a frozen backbone, and full unfreezing. Two Kaggle competitions; due 5/1. Outside readers get the problems but not the course data bundle or tests.
The framework for training from scratch: LitGPT's no-abstraction rewrites, pretrain flow, and the TinyLlama track record
LitGPT (Lightning AI, ~13,600 stars, Apache 2.0) rewrites 20+ mainstream LLMs — Llama 3, Qwen2.5, Phi 4 — from scratch as single-file, no-abstraction implementations, with a full pretrain / finetune / evaluate / serve CLI. TinyLlama (1.1B parameters, 3T tokens) was trained on this codebase. This post breaks down how it differs from MiniMind, how to actually use it, and where it stops.
MiniMind: Train an LLM From Scratch for $0.40
MiniMind is an open-source project for training LLMs from scratch: a 64M Dense model and a 198M-A64M MoE model that run the entire chain — Pretrain → SFT → LoRA → DPO → PPO/GRPO/CISPO → Agentic RL — in ~2 hours on a single RTX 3090 at roughly 3 RMB (~$0.40). Every core algorithm is implemented natively in PyTorch with no high-level wrappers.
Running MiniMind on RunPod: From Zero to a Chatting Model
The hands-on installment of the series: rent an RTX 3090 on RunPod (Secure Cloud $0.5/hr, Community Cloud $0.22/hr), follow the MiniMind README through pretrain (~1.21h) + SFT (~1.10h), spend roughly $0.55–1.50 USD total, and chat with your own 64M model trained from scratch in the terminal.
Training an LLM from Scratch: A Project Spectrum from $0.4 to 65B
Open-source projects have pushed the cost of training an LLM from scratch absurdly low: MiniMind runs the full PreTrain-to-RL pipeline for about $0.4 (2 hours on a single RTX 3090), while at the other end OLMo 3 and LLM360 K2 publish everything — data, code, and stage-by-stage checkpoints of 65B models. This series walks the whole project spectrum from $0.4 to 65B in 11 articles, and flags where the map is biased.
CNN Architectures — From Convolution Kernels to Translation Equivariance | MIT 6.7960 Lecture 4
Lec 4 core: why CNN is the natural choice for grid data — convolution, translation equivariance, pooling, and classic architectures in one go
Course Introduction — A Map of Deep Learning, Why Depth Works, and Your First Training Loop | MIT 6.7960 Lecture 1
Lecture 1 is the 6.7960 opener: deep learning took off because data + compute + algorithms matured together; the course threads from architectures (CNN/GNN/Transformer) through training, representation, generation, transfer, scaling, and LLMs; ends with a ~30-line PyTorch training loop to confirm your environment works.
Modern CNN Architectures — ResNet, EfficientNet, ConvNeXt | MIT 6.7960 Topic Guide
ResNet's skip connections solve degradation, enabling 100+ layer nets; EfficientNet compound scales depth/width/resolution; ConvNeXt absorbs Transformer design to reclaim CV crown.
Optimization Overview — SGD, Adam, LR Schedules & Scaling Rules | MIT 6.7960 Topic Guide
From SGD to Adam: pick the right optimizer and scale LR with batch size using scaling rules
PS1 Walkthrough: From NumPy MLP to PyTorch Autograd Backprop | MIT 6.7960 Assignment 1
Hand-write NumPy MLP + backprop → verify with PyTorch Autograd, fully reproducing OCW HW1 core concepts
Regularization in Practice — Weight Decay, Dropout, Batch Norm & Label Smoothing | MIT 6.7960 Topic Guide
Regularization isn't just anti-overfitting — mechanisms & combo strategies for WD, Dropout, BN, Label Smoothing
Build Data, Tensor, and Pipeline Parallelism from Collectives | Stanford CS336 Lecture 7
Lecture 7 starts below FSDP APIs, building a communication language from broadcast, all-reduce, all-gather, reduce-scatter, and all-to-all before assembling data, tensor, and pipeline parallelism.
Count FLOPs and Memory Before Asking Whether a Model Fits | Stanford CS336 Lecture 2
Lecture 2 reduces model training to tensors, FLOPs, bytes, and time: use einops to track dimensions, arithmetic intensity and roofline analysis to identify bottlenecks, then trade compute for memory with gradient accumulation and activation checkpointing.
Nine Lectures and Three Labs Are Public, but the Full Path Still Uses Three External Services | MIT 6.S191 Course Overview
MIT 6.S191's 2026 edition publishes nine lecture videos, slides, three software labs, and solutions, making it an A3 self-study course. The supplied path still depends on Google/Colab, Comet, and OpenRouter for Lab 3, while unaffiliated learners do not receive MIT credit, project feedback, or API credits.