Scaling laws study model and data allocation under a budget. MoE increases expert capacity while activating selected experts. The former concerns allocation and empirical models; the latter concerns architecture and routing. Both deserve separate checks.
Yueqi Song breaks agent SFT into six decisions: compute loss on assistant tokens only (including the stop token); choose trajectories carefully (runs that pass tests can still teach bad habits, and switching teachers or adding new tasks beats sampling more); unify formats with the Agent Data Protocol; watch packing and template drift during training; evaluate in the real harness; and pick the SFT checkpoint for the RL that follows, not for its own best score.
Snorkel AI raised a $350M Series E co-led by Insight Partners and S32, nearly tripling its valuation from $1.3B (Series D) 17 months ago to $3.5B, with annualized recurring revenue growing 18x in 12 months to $375M. The signal here is that the training bottleneck for frontier models has shifted from 'do we have enough compute' to 'do we have refined enough training data and reinforcement learning environments' — and the data supply chain itself is becoming an infrastructure layer that can raise huge rounds on its own.
Switzerland's Apertus (8B/70B, 15T tokens, 1,000+ languages) filters opt-outs and personal data before training to satisfy the EU AI Act; Japan's LLM-jp consortium shipped LLM-jp-4 (12T tokens) in April 2026, claiming wins over GPT-4o and Qwen3-8B on standard benchmarks. Both prove that from-scratch training outside the English sphere is a data-governance problem, not a technical one — plus a note on RWKV-7 as the non-Transformer alternative.
"Open-source LLM" is a spectrum: weights-only (Llama), weights plus data (most fully open projects), or data order, intermediate checkpoints, and training logs all released (LLM360 K2, OLMo 3's model flow). This piece unpacks the two projects that pushed transparency furthest: OLMo 3 shipped the first fully open 32B thinking model in November 2025, and K2 is the first 65B-class model whose checkpoints even include optimizer states.
Allen AI's OLMo is the only language model family that fully publishes weights, training data (Dolma, 9.3T tokens), training code, all intermediate checkpoints, and evaluation tools. OLMo 3's 32B Think model hits 96.1% on MATH — and you can use OlmoTrace to trace any output back to the exact training data that produced it.
Lecture 14 moves raw documents through language, quality, and safety filtering; exact and near deduplication; and source mixing. Each stage reshapes model behavior, while synthetic instruction and agent trajectories extend the pipeline into executable environments.
Lecture 13 traces training sources through Common Crawl, Wikipedia, GitHub, arXiv, books, and open datasets. Technically accessible is not the same as licensed, and raw data is not training data; provenance must precede cleaning and mixing.
`batch_runner.py` runs thousands of prompts in parallel into ShareGPT-format tool-calling trajectories, lets each prompt name its own container image, and resumes by matching prompt content rather than index. Two quality filters run before you see the data: samples with zero reasoning are discarded, and entries calling hallucinated tool names are dropped at merge time. This is why a research lab builds a personal agent — the agent is the data pipeline.