Skip to content
All tags

#bpe

3 posts

Tokenization: The BPE Algorithm, and Why Chinese Costs More Than English

Models charge by tokens, not characters. The BPE algorithm starts from individual bytes and repeatedly merges the most frequent adjacent pair to build a vocabulary. English 'understanding' might be 1-2 tokens, but Chinese '理解' could take 2-3 — same meaning, higher cost.

CS224N Lecture 14: How Tokenization Creates Multilingual Cost Gaps

Lecture 14 moves from word, character/byte, and subword segmentation to BPE failures and cross-lingual fairness. A tokenizer determines sequence length, compute cost, and the units a model sees; it is not neutral preprocessing.

CS336 Lecture 1: From Bytes to a Tokenizer—and What Deserves to Scale

CS336's first lecture does not treat building a language model from scratch as reenacting every old technique. It separates mechanics, mindset, and intuitions, then uses BPE to show how raw bytes become trainable tokens.