Skip to content
All tags

#multilingual-nlp

1 posts

CS224N Lecture 14: How Tokenization Creates Multilingual Cost Gaps

Lecture 14 moves from word, character/byte, and subword segmentation to BPE failures and cross-lingual fairness. A tokenizer determines sequence length, compute cost, and the units a model sees; it is not neutral preprocessing.