🌏 中文版
Video status: Videos included. Source details
This post covers CS336 Spring 2026 Lecture 14: Data (filtering, deduplication, mixing, synthetic data), taught by Percy Liang on May 13, 2026. Its primary source is the official executable lecture, lecture_14.py.
Lecture 13 finds raw sources. This lecture decides what enters the token stream. Filtering, deduplication, and mixing are called preprocessing, but each defines the distribution a model learns.
Filtering combines several imperfect judgments
Raw web data passes through format parsing, language identification, and main-text extraction before handling short pages, templates, ads, code ratios, repeated symbols, and low-quality prose. Heuristics are cheap and interpretable. Classifiers capture more complex signals but amplify the preferences of their reference dataset.
The video covers toxicity filtering (a classifier trained on the Jigsaw toxic comments data); personal information, adult content, and other policy categories are site additions not covered in the video. False negatives retain risky content; false positives can systematically remove dialects, communities, or sensitive topics. Thresholds require source-level evaluation, not only a global retention rate.
A reliable pipeline preserves filter reasons and scores per document rather than emitting only a cleaned corpus. This supports threshold changes, bias analysis, and tracing back to provenance.
Exact deduplication begins with hashes
Identical documents can be normalized, hashed, grouped, and reduced to one copy. Hashes turn full content into fixed-size keys suitable for MapReduce. The pipeline still needs collision handling, canonical-document selection, and source mappings.
Exact dedup misses copies with changed whitespace, templates, or a few sentences. Near dedup represents documents as n-gram or shingle sets and compares Jaccard similarity. MinHash approximates set similarity with compact signatures, and locality-sensitive hashing creates candidate buckets instead of quadratic all-pairs comparison.
Dedup saves more than tokens. Repetition increases memorization, train/test leakage, and the weight of widely syndicated pages. Excessive dedup can remove legitimate reuse such as quotations, code boilerplate, or translations.
Mixing determines how often each source is seen
Uniform mixing gives every source equal probability. Proportional mixing follows token count. Temperature or α mixing lies between. A small high-quality source can overfit after many epochs; a large web source can drown out rare languages or code under pure proportional sampling.
Epoch caps limit repetitions per source. Regression-based mixing trains small models, learns a mapping from mixture weights to validation or downstream loss, and searches that surface. Small experiments must simulate large-scale epoching; otherwise they overvalue tiny high-quality sources that would be repeated excessively in the target run.
Synthetic data still has provenance
Reasoning data can begin with human or synthetic prompts, sample several teacher responses, then filter by correctness, format, or reward. A larger teacher is not automatically a better teacher; prompt quality, sampling diversity, and verifiable answers matter.
Software-engineering data is harder. Tasks can come from GitHub PRs, model-injected bugs, or full agent trajectories. Repository dependencies, Docker environments, tests, and protection against reading future commits are part of the data. Without a reproducible environment, a trajectory is only plausible-looking text.
An auditable data pipeline
Every stage should emit document ID, input snapshot, program version, scores and reasons, parent IDs, and output hash. Maintain audit samples for random retained and removed documents, each language and source, and threshold boundaries. The mixing manifest records sampling probabilities, epoch caps, and consumed tokens.
Use small-model ablations to compare not only aggregate validation loss but language, code, safety, memorization, and downstream slices. Lecture 14's core lesson is that data quality is not one classifier score; it is a traceable sequence of decisions.
Material fidelity
This lecture has a Spring 2026 schedule entry and a complete executable artifact. This guide follows its filtering, deduplication, mixing, and synthetic-data sections.
Course video sources
The video below is the corresponding Spring 2026 lecture published by Stanford Online. Lecture number and video ID were checked against the official playlist on 2026-10-10.
Content check: verified against the video transcript (2026-10-10): The video is Lecture 14 (Data, day two). The captions cover language ID, quality classifiers (fastText), toxicity filtering, hash and near duplicates (MinHash, Jaccard), mixing, epoch caps, regression-based mixing, teacher-generated reasoning data and synthetic SWE tasks. The captions only discuss toxicity filtering, not PII or adult-content categories, so that sentence now marks those as site additions; the rest was not compared line by line.
Original videos: Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 14: Data
Course and recording entries:
Update Log
- 2026-10-10: Added explicit video status and checked recording sources and access notes.
- 2026-10-10: Rechecked video status. Checked video IDs and lecture numbers against the official playlist; they are correct, and video titles now use the original titles.
- 2026-10-10: Checked the video content against its transcript. The captions only cover toxicity filtering; PII and adult-content categories are now marked as site additions.
References
Loading...