RNN Language Models: Sequence Probabilities and Gradient Paths | CMU 10-423 Lecture 1
A sequence such as “I like cats” is not a bag of independent words.
#language-model
15 postsTransformer, Training, and Decoding: Three Different Questions | CMU 10-423 Topic Guide
How the model reads a history, learns from targets, and chooses its next token are different questions. Attention mixes information; the training objective updates parameters; decoding selects output under a fixed model. Keeping them distinct prevents confusing a sampling change with a model update.
Discrete Diffusion, Generating Language with CTMCs | MIT 6.S184 Lecture 5
Text is a sequence of discrete tokens. There is no direction to move in, so ODEs and SDEs do not exist. Lecture 5 carries the recipe from Lectures 1–4 over unchanged and swaps only the underlying stochastic process: vector fields become rate matrices, ODEs become continuous-time Markov chains (CTMCs), and the continuity equation becomes the Kolmogorov forward equation. With the factorized mixture path, the marginal rate matrix has exactly one unknown: the probability of each position's original token given the noisy sequence. Training a discrete diffusion model therefore reduces to per-position classification with a cross-entropy loss. Make the noise all [mask] tokens and you get a masked diffusion language model.
Word Embeddings and Language Models, from Counting N-grams to RNNs | NTHU Hung-Yu Kao Natural Language Processing Week 2
Week 2 of Hung-Yu Kao's NLP course at NTHU is a 62-page deck about predicting the next word. The first part covers statistical language models with a bigram count table, add-one smoothing, and perplexity, then sparse vectors through a PPMI example with cherry and digital. The middle returns to Word2Vec's negative sampling and uses the Chinese word for 'apple' to show why contextualized embeddings are needed. The last part derives RNNs from three weaknesses of feedforward networks and shows how they handle NER, sentence classification, and stacked and bidirectional variants.
Word Vectors, Language Models, and RNNs: How to Represent a Word and Predict the Next One | NTU Yun-Nung Chen Applied Deep Learning Lecture 3
This lecture has one through-line: a language model estimates the probability of a word sequence by predicting the next word one step at a time. It first covers how to represent a word, from dictionaries, one-hot vectors, and co-occurrence matrices to low-dimensional word embeddings. It then covers three generations of language models: n-grams count, and unseen combinations get zero probability; a neural LM predicts with a network, and similar words share probability automatically; an RNN LM accumulates all previous words in a hidden state and shares one set of weights across time steps. RNNs are trained with BPTT, multiplying by the same matrix many times makes gradients vanish or explode, clipping handles explosion, and the gating in LSTM and GRU handles vanishing. Finally, applications are divided by the shape of input and output into whole-sequence classification, tagging, and encoder-decoder.
Three Kinds of Noise, One Training Objective, and the Price of Parallel Decoding in Diffusion LLMs | Stanford CME295 2026 Edition Lecture 8
The 2026 edition of CME295 gives diffusion LLMs a full lecture (Lecture 8, November 20), with five listed subtopics: continuous, discrete and masked diffusion, training, and inference. This pre-lecture edition works from the original papers (DDPM, D3PM, SEDD, MDLM, LLaDA and others): continuous noise costs about 64x the compute on text, and the [MASK] absorbing state won out; the training objective is a masked cross-entropy weighted by 1/t; the speed comes from filling several positions per step, yet LLaDA's main results decode one token per step, and Fast-dLLM needs a confidence threshold plus an approximate KV cache to reach up to a 27.6x speedup.
Language — N-grams, Parser, Attention & the Transformer | Harvard CS50 AI Week 6
Week 6 is about getting computers to handle natural language: parse sentences with hand-written context-free grammar rules, then switch to statistics, from n-grams and Markov-chain text generation to a Naive Bayes classifier for review sentiment; turn words into vectors with word2vec, process whole sequences with an RNN encoder–decoder, and finish with attention and the Transformer. Includes section-by-section time links, plus an approach, common mistakes and ways to check your work for the Parser and Attention projects.
From Tokenization to N-gram Language Models | CMU 07-280 Lecture 17
Lecture 17 first decides how text becomes tokens, then uses N-grams to turn sequence probability into conditional probabilities estimated from corpus counts. Tokenization is the first design decision about what a model can see.
How N-grams Train, Sample, and Fail | CMU 07-280 Lecture 18
Lecture 18 truncates the chain rule with an N-gram Markov assumption, estimates probabilities from corpus counts, and contrasts greedy, categorical, and temperature sampling. The real bottlenecks are zero probability for unseen contexts and a fixed window.
Turning Next-token Prediction into Geometry | CMU 07-280 Lecture 19
Lecture 19 builds a minimal next-token model from two embedding matrices, dot-product similarity, softmax, and cross-entropy. Shared vector parameters replace the isolated count cells of an N-gram table.
Words, Tokens, Edit Distance, and N-grams: Decide What the Model Sees First | Stanford CS124 Week 2
Week 2 teaches three things: cutting text into tokens with BPE, computing the edit distance between two strings with dynamic programming, and estimating the next word's probability with n-grams, evaluated by perplexity. Each can be done on paper, and this post works one example of each.
Pretraining, Subwords, and In-Context Learning | Stanford CS224N Lecture 7
Lecture 7 decomposes pretraining into scalable data, subword tokenization, three model objectives, and in-context learning. A general self-supervised objective yields reusable representations; downstream signals specify their use.
Language Models, RNNs, and Vanishing Gradients | Stanford CS224N Lecture 4
Lecture 4 defines a language model as a next-word probability distribution, then uses an RNN to compress an arbitrarily long prefix. It also exposes recurrence's central cost: information and gradients travel one time step at a time.
Scaling Language Models When Data Is the Bottleneck | Stanford CS224V Lecture 14
The final lecture is not a complete LLM-training tutorial. It studies data efficiency under fixed data and abundant compute, revisiting epochs, batches, ensembles, self-training, and conditions for synthetic continued pretraining.
From N-grams and Word Representations to Text Classification | Berkeley CS288 Topic Guide
The first four units make text countable, representable, and classifiable; A1 then moves from n-grams and perceptrons to an NBOW MLP.