🌏 中文版
Video status: Related supplementary video included; the original lecture recording has not been verified. Source details
The official CS224N Winter 2026 schedule places lecture 11 on February 10, 2026, but does not name a lecturer; this article therefore attributes it only to the course staff. The official deck has four agenda parts: the recent saga of LLM benchmarks, benchmark design, evaluation metrics, and cautions and open questions. The task is not to memorize another leaderboard but to decide whether a score still supports a decision.
Benchmark shelf lives are shrinking
The NLP benchmarking survey explains how benchmarks and leaderboards turn research questions into shared targets. When models approach saturation, scores lose discriminative power. Test data may also enter pretraining corpora, turning generalization into recall. Human performance is not always a meaningful ceiling when a task rewards broad retrieval, strict formatting, or speed rather than human expertise.
Results therefore need model version, prompt, decoding settings, and evaluation date. A benchmark name alone is not reproducible.
What to test: static, dynamic, and adversarial design
A high-impact benchmark needs a clear construct, representative data, and enough difficulty to distinguish methods. Static test sets are comparable but prone to saturation and leakage. Dynamic benchmarks refresh data. Adversarial benchmarks target weaknesses, though collection can introduce annotation artifacts that models exploit as shortcuts.
Multitask evaluation broadens coverage, but an average can hide subgroup failures. A leaderboard should be expanded into tasks and categories rather than read only by its aggregate.
How to measure: references, models, and people
HELM demonstrates holistic evaluation across scenarios and metrics. Exact match is cheap and reproducible but too strict for open generation; model-based metrics such as AlpacaEval handle semantics and preferences but inherit position bias, verbosity bias, version drift, and knowledge gaps from the evaluator.
Human evaluation is not automatically a gold standard. A vague rubric, mismatched annotator expertise, or low agreement also produces unstable results. Preserve the rubric, samples, randomization, and agreement statistics.
Goodhart, contamination, and prompt sensitivity
Once a metric becomes an optimization target, it may cease to represent the original quality. Contamination puts test items into training; prompt sensitivity changes measured capability with formatting. Trustworthy evaluation needs contamination audits, multiple reasonable prompts, error analysis, and a holdout not repeatedly used for development.
Write an evaluation contract
Fix construct, population, inputs/outputs, metric, prompts, decoding, versions, date, and exclusions before test results. A task format is not automatically the capability name.
Benchmark lifecycle
Track collection, annotation, release, leaderboard probing, saturation, and eventual training contamination. Version hashes and combine stable anchors with rolling fresh sets.
Construct validity and shortcuts
Use counterfactuals and simple input-only baselines to detect artifacts, memorization, and format cues. Human ceilings require matched tools, time, and expertise.
Dynamic and adversarial evaluation
Dynamic sets improve freshness but drift; adversarial sets expose weaknesses but create collector artifacts. Report natural, adversarial, and dynamic distributions separately.
Aggregating multitask benchmarks
Macro, micro, normalized scores, and ranks encode different weights. Publish per-task, uncertainty, and worst-group results; count refusals by a predefined rule.
Reference-based metrics
Exact match, token F1, BLEU, ROUGE, and learned semantic metrics measure different overlap. Validate correlation in the target domain and preserve normalization and references.
Reference-free and information-theoretic metrics
Split rubrics so fluency cannot replace correctness. Calibration, likelihood, and uncertainty need tokenizer/API-aware interpretation and reliability plots.
Human evaluation protocol
Train qualified annotators with anchors, blind identity, randomize order, double-label a subset, report agreement, and size samples for expected differences.
Bias tests for LLM judges
Lock judge versions and test order, verbosity, self-preference, evidence use, and prompt injection. Audit with humans or independent judges.
Contamination audit
Check exact, near-duplicate, paraphrase, template, and answer leakage. Use temporal/canary evidence when training data is closed and state uncertainty.
Prompt sensitivity and uncertainty
Treat reasonable prompts and seeds as variation; use paired/bootstrap intervals and distinguish statistical from practical significance.
An executable checklist
Document construct through cost, classify twenty errors, rerun prompt paraphrases and seeds, swap judge order, and state explicitly what the score cannot prove.
Material gap
Winter 2026 recordings are not public. This article covers the deck's four agenda sections and subtopics but does not reconstruct the lecturer's spoken judgments about particular leaderboards.
Course video sources
This article uses Winter 2026 materials. The Spring 2024 recording(s) below cover the same topic as supplementary material and are not Winter 2026 lecture recordings (Winter 2026 recordings are on Canvas behind sign-in, and the course says they are not open to non-enrolled students).
Original videos: Stanford CS224N: NLP with Deep Learning | Spring 2024 | Lecture 11 - Benchmarking by Yann Dubois
Course and recording entries:
Checked: 2026-10-10.
Content check: verified against the video transcript (2026-10-10): what was checked is how the Spring 2024 Lecture 11 (about 84 minutes; the YouTube title credits Yann Dubois, but the transcript itself never names the speaker) relates to this article, which is based on the Winter 2026 slides; it is not the 2026 classroom recording. The transcript covers the overall evaluation framework, close-ended tasks (multiple choice, MMLU, SuperGLUE and multi-task averaging), open-ended evaluation (reference-based metrics, human evaluation and annotator selection, LLM-as-judge such as AlpacaEval), perplexity, robustness of rankings to small prompt changes, contamination and dynamic test sets, matching this article's topic. It never mentions Goodhart or construct validity and has no adversarial design or concrete procedures such as an "evaluation contract" or "contamination audit", so those sections rest on the Winter 2026 slides and this site's own additions, not on the video.
Update Log
- 2026-10-10: Added explicit video status and checked recording sources and access notes.
- 2026-10-10: Rechecked video status. Switched to related supplementary video: embedded the Spring 2024 public recording on the same topic; the Winter 2026 original is behind sign-in and could not be verified.
- 2026-10-10: Checked the video content against its transcript. The 2024 video covers close/open-ended evaluation, LLM judges, prompt robustness, contamination and dynamic test sets; Goodhart, construct validity, adversarial design and audit procedures are not in the video and are marked as not video content.
References
Loading...