Lecture 5 turns classification around: instead of learning p(y|x) directly, it learns p(x|y) and p(y). GDA assumes the two classes are Gaussians with a shared covariance. Maximum likelihood then takes nothing more than counting proportions and averaging, the decision boundary comes out as a straight line, and the posterior is exactly a sigmoid. Naive Bayes uses conditional independence to squeeze 2^d parameters down to 2d+1.
This lecture splits test error into three terms: noise σ², squared bias, and variance. It then uses the closed-form solution of ridge regression to show how regularization moves each eigenvalue from λ to λ+ρ, trading a little bias for a large cut in variance. The second half covers double descent, ImageNet-V2, and choosing hyperparameters with a dev set, k-fold cross-validation, and successive halving.
Lecture 7 first carries the training recipe for linear models over to nonlinear models unchanged: one loss per example, take the average, walk downhill with SGD. Two questions remain: what the model looks like, and how to compute the gradient. The second half answers the first. One neuron is ReLU(wᵀx+b); stacking m neurons into a matrix gives one layer, ReLU(Wx+b), with md+m parameters; above that come the residual block z+σ(MM(σ(MM(z)))) and LayerNorm and RMSNorm. Computing the gradient is left to Lecture 8.
The lecture starts from one theorem: if a scalar function can be computed with N basic operations, its gradient can also be computed in O(N) time. The method is to read the chain rule as "given ∂J/∂u, you can compute ∂J/∂z, using only information about the module in between", then derive a backward function for matrix multiplication and for the activation function: Wᵀv, vzᵀ, and σ′(z)⊙v. This article also works a forward and backward pass by hand on a small 2-2-1 network.
Lecture 9 starts from the two steps of k-means, assign and average, and explains why the algorithm always stops yet only reaches a local optimum. Replacing hard assignments with probabilities gives the Gaussian mixture model and the two steps of EM: an E-step that is Bayes' rule and an M-step that is a weighted average. The lecture ends with the picture behind EM, drawn with Jensen's inequality: each round builds a lower bound that touches the log-likelihood and climbs to its top. The full derivation waits for Lecture 10.
The first half uses Jensen's inequality to build a lower bound on the log-likelihood, the ELBO, then picks Q to be the posterior so the bound is tight at the current parameters. From that it follows that no round of EM can lower the likelihood, and plugging in a Gaussian mixture recovers last lecture's weighted-average formulas. The second half writes PCA as maximizing uᵀΣu over unit vectors; the answer is the eigenvector of the covariance matrix with the largest eigenvalue.
Each forward step multiplies the data by √(1−β_t) and adds Gaussian noise with variance β_t. Unrolled, x_t = √ᾱ_t·x_0 + √(1−ᾱ_t)·ε̂_t, and nothing here is learned. The reverse direction is learned with a Gaussian p_θ(x_{t−1}|x_t): apply last lecture's ELBO to the whole trajectory, split it with the KL chain rule, and the loss at each step is ‖μ̃_t(x_t,x_0) − μ_θ(x_t,t)‖²/(2β̃_t). The lecture stops at this expression; simplifying it into noise prediction is left for Lecture 12.
The first half collapses the diffusion ELBO into one expression: every time step is a squared error for guessing the noise that was added, which gives a five-step training procedure and a one-loop sampler. The second half opens the foundation-model unit: the two-stage paradigm of pretraining plus adaptation, linear probe, finetune, LP-FT, and why LoRA trains only 2dr parameters yet saves very little compute.
This lecture unpacks the SimCLR loss into a B×B similarity matrix: each row is one softmax classification problem whose answer is the diagonal entry. The diagonal gets pulled up and everything else gets pushed down, so the model learns that similar things have nearby vectors without any labels. The second half uses the same vectors for semantic retrieval, then connects that to RAG.
This lecture takes a large language model apart in three layers: tokenization turns text into token IDs, the chain rule splits the probability of a whole text into a sequence of next-token conditionals, and training sums the cross-entropy at every position. The second half opens up the Transformer: inner products of queries and keys go through a softmax to give weights, the weights average the values, and a causal mask blocks future positions. The cost of a single attention layer grows with the square of the sequence length T.
During generation, every head has to keep the keys and values of all past positions in GPU memory, and this KV cache caps the batch size. GQA lets τ query heads share one set of keys and values, shrinking the cache to 1/τ; sliding window attention cuts the T² compute down to T·w; MoE runs only 8 of 128 experts for each token. The second half covers how to use a trained model: few-shot and zero-shot prompting leave the parameters alone, SFT changes them, and its loss is computed only on the answer tokens.
The first 55 minutes of this lecture write sequential decision making as an MDP (states, actions, transition probabilities, rewards, discount factor) and show that the Bellman equation for a fixed policy is a system of linear equations. The last 20 minutes derive policy gradient: ∇p = p∇log p turns the gradient back into an expectation, the transition probabilities drop out of the gradient because they don't contain θ, and the lecture stops at the formula "sum of log-probability gradients × sum of discounted rewards", leaving variance reduction for the next lecture.
The last lecture of the term finishes policy gradient: past rewards and any baseline that depends only on the state can be removed from the weight without changing the expectation. It then derives PPO's clipped objective from importance sampling, splits it into four cases to see which samples still carry a gradient, and compares the change CISPO makes. Finally it writes token-by-token generation in a language model as an MDP, gives a reward of 1 only when the answer is right, and trains long chains of thought using the average over 8 samples of the same question as the baseline.
In a 36-minute video, Dan Koe argues people with many interests should split skills into human nature, domain, and tool layers, hand only the tool layer to AI, and treat AI output as a first draft.
A chert crag in Shimukappu village, central Hokkaido. ROCKTOPO lists 113 lines in total (101 face routes, 11 boulder problems), with an unusually deep 5.12–5.13 range, and the walls sit right beside Prefectural Road 136. Season is May–October; winter snow reaches about 1 m. The nearest station is Shimukappu on the JR Sekisho Line, but no official bus serves the crag, so in practice you drive.
A sandstone sport crag in Joban-Yumoto, Iwaki City, Fukushima. ROCKTOPO lists 100 lines (85 sport, 15 boulders); most routes are only 5–6 m and the 5.10–5.11 range dominates. Both crag databases give a September–May season, with sunny winters as the main draw. The nearest station is JR Yumoto, but this is a driving crag with about 13 parking spaces below the walls.
A cluster of twenty-plus limestone crags around Takahashi, Okayama. ROCKTOPO lists 347 routes, almost all bolted sport lines, concentrated in the 5.11–5.12 range. Something is climbable year-round: shady valleys in summer, south-facing walls in winter. The base at Yuze hut is about 30 minutes by bus from JR Bitchu-Takahashi Station.
A tuff sport crag in the Ura-Rokko area of Kita Ward, Kobe, Hyogo. CLIMBING-net lists 70+ routes and ROCKTOPO 94 (only 5 at 5.12 or harder); both list it as climbable year-round. It is a 15-minute walk from JR Dojo Station on the Fukuchiyama Line. Many bolts are old, the rock is not great, and parking is about five roadside spots.
A limestone sport crag in Ogano, Saitama, with walls at 90–135 degrees. CLIMBING-net lists 160 routes and ROCKTOPO 220, concentrated in the 5.11 and 5.12 range. CLIMBING-net calls it a year-round crag but notes seepage in the rainy season and summer. By transit it takes two buses from Seibu-Chichibu Station to Sakamoto plus a hike up to Matatoge; most climbers drive up the forest road.