Table of Contents
🌏 中文版
The official CS224N Winter 2026 schedule confirms that the seventeenth regular unit took place on March 3, 2026, guest-taught by Luke Zettlemoyer under Guest Lecture: Multimodality. The course page publishes no current slides or agenda and lists four suggested plus seven optional readings. This article covers only the five public sources actually read and listed below; it does not present the other six as read or reconstruct the session.
Vision as a reasoning workspace
Visual Sketchpad puts drawing, marking, and visual tools inside a reasoning loop. The model can create an intermediate visual state and inspect it to continue reasoning. Unlike text-only chain of thought, this external workspace can carry spatial relationships.
Evaluation must separate tool capability from the model's decision about when to use it. A successful tool call does not imply reliable planning.
One token stream for multiple modalities
Chameleon uses mixed-modal early fusion, placing text and image representations in one sequence for a single autoregressive model. Mixture-of-Transformers explores sparse modality-specialized parameters within a shared sequence architecture.
A unified interface reduces task-switching friction but does not prove that every modality should share identical representations or losses. Image compression, token count, and continuous detail change the cost structure.
Different modalities can keep different objectives
Transfusion applies next-token prediction to discrete text and diffusion loss to continuous images in one model. It avoids forcing all images into discrete tokens, at the price of a more complex training and sampling pipeline.
The optional subset actually read here contains only Multimodal RewardBench. It shows why multimodal evaluation should separate general capability from safety preference and test whether a reward model behaves consistently when text, images, or both carry the decisive signal. The other optional readings listed by the schedule are outside this article's summary scope.
What can and cannot be confirmed
The date, speaker, title, and the fact that the official page lists eleven readings are confirmed. This article actually reads and summarizes the five sources in its reference list; the other six appear only in the schedule and support no content claim here. The actual agenda, architecture comparisons, live demonstrations, and conclusions are not confirmed. Reading-list order is not presented as teaching order. A slide-by-slide review should replace this map only if current slides are released.
References
Loading...