RNN Language Models: Sequence Probabilities and Gradient Paths | CMU 10-423 Lecture 1
A sequence such as “I like cats” is not a bag of independent words.
#rnn
8 postsRecurrent Neural Networks and Image Captioning — RNNs, LSTMs, and Captioning | Stanford CS231N Lecture 7
An RNN updates one hidden state at every time step with the same weights, so it can handle sequences of any length. CS231N Lecture 7 starts by hand-building an RNN that detects repeated 1s. It then covers character-level language models and feeding CNN features into an RNN for image captioning. Gradient flow explains why vanilla RNNs are hard to train: clip gradients to stop them exploding, and change the architecture (the LSTM) to stop them vanishing. The lecture ends by calling state space models like Mamba "modern RNNs."
Arithmetic as a Language — PyTorch TA Session, a Two-Layer LSTM, and Teacher Forcing | NTHU Hung-Yu Kao Natural Language Processing Assignment 2
HW2 treats expressions like "14*(43+20)=882" as character sequences and asks a two-layer LSTM to generate the answer one character at a time after it sees "=". The training set has 2,369,250 rows and the eval set 263,250, with every number in 0–49. Six TODOs run from building a vocabulary and batching with loss only after "=" to a generator, teacher-forced training, and exact-match evaluation. The W4 PyTorch TA session is the toolbox for it.
NTHU NLP W3: When Input and Output Lengths Differ — Seq2seq, LSTM, and Attention | NTHU Hung-Yu Kao Natural Language Processing Topic Guide
"Look over there" is 3 tokens, the Chinese version 4, the Japanese version 12. When output length does not track input length, the classifier trick of adding an FFN at the end fails. This 33-slide deck starts from encoder-decoder models, works through vanishing gradients in RNNs and the three LSTM gates, and ends with attention fixing both long-range memory and parallelism, including why scores are divided by √d.
Word Embeddings and Language Models, from Counting N-grams to RNNs | NTHU Hung-Yu Kao Natural Language Processing Week 2
Week 2 of Hung-Yu Kao's NLP course at NTHU is a 62-page deck about predicting the next word. The first part covers statistical language models with a bigram count table, add-one smoothing, and perplexity, then sparse vectors through a PPMI example with cherry and digital. The middle returns to Word2Vec's negative sampling and uses the Chinese word for 'apple' to show why contextualized embeddings are needed. The last part derives RNNs from three weaknesses of feedforward networks and shows how they handle NER, sentence classification, and stacked and bidirectional variants.
Word Vectors, Language Models, and RNNs: How to Represent a Word and Predict the Next One | NTU Yun-Nung Chen Applied Deep Learning Lecture 3
This lecture has one through-line: a language model estimates the probability of a word sequence by predicting the next word one step at a time. It first covers how to represent a word, from dictionaries, one-hot vectors, and co-occurrence matrices to low-dimensional word embeddings. It then covers three generations of language models: n-grams count, and unseen combinations get zero probability; a neural LM predicts with a network, and similar words share probability automatically; an RNN LM accumulates all previous words in a hidden state and shares one set of weights across time steps. RNNs are trained with BPTT, multiplying by the same matrix many times makes gradients vanish or explode, clipping handles explosion, and the gating in LSTM and GRU handles vanishing. Finally, applications are divided by the shape of input and output into whole-sequence classification, tagging, and encoder-decoder.
Memory and Sequence Modeling — RNNs, LSTMs, and Vanishing/Exploding Gradients | MIT 6.7960 Lecture 10
RNNs propagate hidden states through time; gradient products can shrink or grow. LSTM gates control cell-state retention and updates, improving long-range signal flow without guaranteeing stability. Self-attention shortens paths between positions, while dense attention still has quadratic sequence-length cost.
Language Models, RNNs, and Vanishing Gradients | Stanford CS224N Lecture 4
Lecture 4 defines a language model as a next-word probability distribution, then uses an RNN to compress an arbitrarily long prefix. It also exposes recurrence's central cost: information and gradients travel one time step at a time.