🌏 中文版
Video status: Related supplementary video included; the original lecture recording has not been verified. Source details
The official CS224N Winter 2026 schedule places lecture 2 on January 8, 2026, but does not name a lecturer; this article therefore attributes it only to the course staff. The official deck covers word2vec, objective-function gradients, optimization basics, count-based representations, and evaluation, after brief course logistics. Its concrete goal is to understand word meaning as a high-dimensional real vector and to read embedding papers.
From dictionary nodes to positions in context
Lexical resources such as WordNet organize words into synonym sets and hypernym relations. Their structure is explicit, but they require manual maintenance, miss new uses, and do not naturally quantify degrees of similarity. Distributional representation begins elsewhere: words occurring in similar contexts should receive similar representations.
word2vec assigns vectors to words and predicts surrounding words from a center word. A sliding window supplies center–context pairs; training adjusts the vectors so observed pairs receive higher probability. “Meaning” becomes geometry that a learning objective can manipulate.
What the objective and gradient do
The slides develop the skip-gram softmax objective. A center vector is scored against possible outside-word vectors and normalized into probabilities. Negative log likelihood penalizes the model when the observed context is not ranked highly. The gradient has a useful intuition: predicted distribution minus observed distribution. Overpredicted candidates move away; the observed context moves closer.
A full softmax scans the vocabulary for every pair. The official reading list therefore introduces negative sampling: combine each observed pair with a small number of negative examples and replace multiclass prediction with several binary decisions. This changes the training objective, so it is more than a harmless numerical shortcut.
Prediction is not the only road: count first
The agenda then returns to co-occurrence matrices. One can count word–word or word–document events across a corpus and compress the matrix with a method such as SVD. Predictive methods update parameters example by example; count-based methods begin with global statistics. GloVe connects the two by learning from ratios of word-pair co-occurrences.
Both routes share the distributional assumption that corpus neighborhoods carry semantics. They also share limitations: rare words have little evidence, one vector conflates multiple senses, and stereotyped relationships in the corpus remain in the geometry.
Evaluation: a neat analogy is not downstream success
The slides distinguish intrinsic and extrinsic evaluation. Intrinsic tests directly measure similarity, analogies, or an intermediate property; they are cheap and inspectable. Extrinsic tests insert the representation into a real task and measure system performance, which is closer to use but makes component-level attribution difficult.
When reading an embedding result, ask whether the benchmark evaluates the vector itself or a system containing many other decisions. A high score on one benchmark does not prove that a vector captures “meaning itself.”
How skip-gram creates training pairs
A context window turns a sentence into center–outside pairs. Window size, distance weighting, subsampling, punctuation, tokenization, dimensions, negative samples, and training steps all affect geometry. One static vector also merges senses, weighting them by corpus frequency.
From scores to softmax
For center (c) and outside word (o):
[ P(o\mid c)=\frac{\exp(u_o^T v_c)}{\sum_{w\in V}\exp(u_w^T v_c)}. ]
The loss forces the observed context to outrank the vocabulary. Its gradient has the expected-minus-observed form reused throughout classifiers and language models.
Negative sampling
The negative sampling paper replaces a vocabulary-wide softmax with one positive and (K) noise pairs. The noise distribution is part of the method. A sampled negative is not an antonym; it is merely unobserved in that training window.
Count vectors, PMI, and SVD
Count methods build a target-by-context matrix, correct raw frequency with PMI/PPMI, and compress it with truncated SVD. GloVe instead learns dot products from weighted log co-occurrence counts. Both routes compress distributional evidence.
What embedding geometry cannot prove
Neighbors and analogies depend on preprocessing and frequency. Hubness and projection artifacts can make plots misleading. Corpus associations also preserve social bias; removing one linear direction does not prove downstream harms disappeared.
A lecture-aligned notebook exercise
The public Assignment 1 notebook supports co-occurrence, SVD, similarity, and bias experiments. Change one setting at a time and record neighbors for fixed queries. Compare an intrinsic score with a fixed downstream classifier, then inspect a polysemous word to see how one vector compromises between senses.
Static vectors and modern embedding tables
A Transformer still begins with a token embedding table, but contextual layers turn each occurrence into a different hidden state. Comparing “embeddings” therefore requires layer, pooling, normalization, and subword-to-word alignment. Input-table neighbors can reflect spelling and frequency rather than sentence meaning.
How to read an embedding paper
Identify the representation unit, objective and negative evidence, corpus and preprocessing, and intrinsic versus downstream evaluation. Check dimensions, extra data, tuning, and OOV policy. Rewrite each metric as the concrete question it asks rather than circling the largest number.
The boundary of the distributional hypothesis
Corpus context supports relatedness, not every semantic relation. Antonyms often share syntactic frames and become neighbors. Similarity, relatedness, substitutability, and entailment are different targets; one cosine score cannot represent them all.
Material gap
Winter 2026 recordings are not public. This account uses the Lecture 2 deck and the core papers on the official reading list. Classroom demonstrations, spoken derivations, and questions cannot be verified and are not reconstructed.
Course video sources
This article uses Winter 2026 materials. The Spring 2024 recording(s) below cover the same topic as supplementary material and are not Winter 2026 lecture recordings (Winter 2026 recordings are on Canvas behind sign-in, and the course says they are not open to non-enrolled students).
Original videos:
- Stanford CS224N: NLP with Deep Learning | Spring 2024 | Lecture 1 - Intro and Word Vectors
- Stanford CS224N: NLP with Deep Learning | Spring 2024 | Lecture 2 - Word Vectors and Language Models
Course and recording entries:
Checked: 2026-10-10.
Content check: verified against the video transcript (2026-10-10): both transcripts were read, and what was checked is how the Spring 2024 Lectures 1 and 2 (about 80 and 79 minutes) relate to this article, which is based on the Winter 2026 Lecture 2 slides; they are not the 2026 classroom recordings. Lecture 1 opens with course logistics and a discussion of language and word meaning (WordNet, the drawbacks of one-hot vectors, distributional semantics) and from about 69% moves into the Word2Vec objective, the window, softmax and the gradient derivation, matching this article's first two sections. Lecture 2 first reviews gradient descent and SGD, then skip-gram and CBOW and negative sampling, demos analogies with GloVe vectors in a notebook (about 15 to 30%), covers co-occurrence matrices and SVD around 45 to 50%, then the GloVe model, evaluating word vectors (intrinsic versus extrinsic, human similarity judgments) and polysemy (about 73 to 80%), and ends by turning to neural-network classification, matching the article's negative sampling, count-based/SVD and evaluation sections. The transcripts contain no PMI derivation and nothing on how word vectors relate to modern embedding tables, how to read an embedding paper or the specific notebook exercises, so those sections rest on the Winter 2026 slides and this site's own additions, not on the videos.
Update Log
- 2026-10-10: Added explicit video status and checked recording sources and access notes.
- 2026-10-10: Rechecked video status. Switched to related supplementary video: embedded the Spring 2024 public recording on the same topic; the Winter 2026 original is behind sign-in and could not be verified.
- 2026-10-10: Checked the video content against its transcript. Spring 2024 Lectures 1 and 2 cover the Word2Vec objective, negative sampling, co-occurrence/SVD, GloVe, evaluation and polysemy, matching the article's main line; PMI and modern embedding tables are not in the videos and are marked as not video content.
References
Loading...