Scaling laws study model and data allocation under a budget. MoE increases expert capacity while activating selected experts. The former concerns allocation and empirical models; the latter concerns architecture and routing. Both deserve separate checks.
CME295 Lecture 3 defines an LLM as a decoder-only next-token predictor, uses MoE to explain why a huge model only touches part of its weights per token, and spends most of its time on the knobs you can turn at generation time: greedy, beam search, top-k, top-p, temperature, guided decoding, plus three prompting techniques (few-shot, chain of thought, self-consistency). The 2026 edition folds this lecture into Lecture 2, prompting is no longer a separate syllabus item but still gets a brief segment in the recording.
HW4 Problem 3 quantifies why voting trees get more accurate in three steps: with p=0.6 the Hoeffding bound needs B≈691 independent trees to reach 10⁻⁶; correlation ρ between trees floors ensemble variance at ρσ²; and random forests' dense ensembling is compared with MoE's sparse routing.
Stanford Marin pre-registers a paloma macro-loss of 2.04 with a 5-rung Scaling Ladder at 1% cost, then trains 535B-A23B on 11×GB200 in public with live W&B telemetry — 847 training buckets already show the most teachable frontier run.
Lecture 4 studies two kinds of sparsity: linear/recurrent attention reduces sequence-length cost, while MoE activates only part of a model for each token. Both turn saved FLOPs into routing, balancing, communication, and kernel problems.