This lecture has two halves. The first covers tokenization: with whole words as units, unseen words can only become UNK; with characters as units, meaning cannot be put back together. The compromise is subwords, and a common way to build them is BPE: start from characters and repeatedly merge the most frequent adjacent pair. The second half covers BERT: a static word vector gives "apple" one embedding and cannot tell the fruit from the company. ELMo uses a language model to produce contextual representations, and BERT replaces the LSTM with a Transformer, pre-trains with a masked LM that hides 15% of tokens plus next sentence prediction, and then fine-tunes the whole model on downstream tasks.
Pre-training lets a model learn general abilities from large unlabeled data before the task is known. There are three architectures: encoders see both directions and suit understanding; decoders see only the left, can generate, and are the type that won; encoder-decoders pre-train by denoising. Pre-training plus fine-tuning hits two obstacles, scarce labels and oversized models. GPT-3's in-context learning performs tasks with no parameter update, followed by hard prompts (a template plus a verbalizer, LM-BFF) and soft prompts that train vectors directly (P-Tuning, Prefix-Tuning). Soft prompts did better at the time, but what survived is prompting with no training at all.
Transfer learning's core insight is 'features learned on big data are good general-purpose representations': freeze the backbone and train only a linear head when downstream data is tiny; full fine-tune when data is plentiful; reach for LoRA / adapter when compute is tight. SimCLR and MAE removed the need for upstream labels and pushed downstream quality another notch.
Lecture 7 decomposes pretraining into scalable data, subword tokenization, three model objectives, and in-context learning. A general self-supervised objective yields reusable representations; downstream signals specify their use.