Allen AI's OLMo is the only language model family that fully publishes weights, training data (Dolma, 9.3T tokens), training code, all intermediate checkpoints, and evaluation tools. OLMo 3's 32B Think model hits 96.1% on MATH — and you can use OlmoTrace to trace any output back to the exact training data that produced it.
Lecture 14 moves raw documents through language, quality, and safety filtering; exact and near deduplication; and source mixing. Each stage reshapes model behavior, while synthetic instruction and agent trajectories extend the pipeline into executable environments.
Lecture 13 traces training sources through Common Crawl, Wikipedia, GitHub, arXiv, books, and open datasets. Technically accessible is not the same as licensed, and raw data is not training data; provenance must precede cleaning and mixing.
`batch_runner.py` runs thousands of prompts in parallel into ShareGPT-format tool-calling trajectories, lets each prompt name its own container image, and resumes by matching prompt content rather than index. Two quality filters run before you see the data: samples with zero reasoning are discarded, and entries calling hallucinated tool names are dropped at merge time. This is why a research lab builds a personal agent — the agent is the data pipeline.