A guide to the Sub-word Tokenization unit of Prof. Hung-Yu Kao's NLP course at NTHU (Fall 2025). Splitting on white space only works for Western languages and cannot handle unseen words or German-style compounds. BPE starts from characters and repeatedly merges the most frequent adjacent pair into the vocabulary, so each merge adds one entry; its weakness is that the greedy split is not always the best one. The Unigram LM picks splits by probability and can even sample different ones. The slides give vocabulary sizes of 30522 for BERT, 50257 for GPT-2/GPT-3 and 32,128 for T5.
This lecture has two halves. The first covers tokenization: with whole words as units, unseen words can only become UNK; with characters as units, meaning cannot be put back together. The compromise is subwords, and a common way to build them is BPE: start from characters and repeatedly merge the most frequent adjacent pair. The second half covers BERT: a static word vector gives "apple" one embedding and cannot tell the fruit from the company. ELMo uses a language model to produce contextual representations, and BERT replaces the LSTM with a Transformer, pre-trains with a masked LM that hides 15% of tokens plus next sentence prediction, and then fine-tunes the whole model on downstream tasks.
Models charge by tokens, not characters. The BPE algorithm starts from individual bytes and repeatedly merges the most frequent adjacent pair to build a vocabulary. English 'understanding' might be 1-2 tokens, but Chinese '理解' could take 2-3 — same meaning, higher cost.
Lecture 14 moves from word, character/byte, and subword segmentation to BPE failures and cross-lingual fairness. A tokenizer determines sequence length, compute cost, and the units a model sees; it is not neutral preprocessing.
CS336's first lecture does not treat building a language model from scratch as reenacting every old technique. It separates mechanics, mindset, and intuitions, then uses BPE to show how raw bytes become trainable tokens.