Transformers and Attention: How Models Decide Which Words to Look At

🌏 中文版

The Scene: What Does "It" Refer To?

Read this sentence:

The cat sat on the mat because it was tired.

You know "it" refers to the cat, not the mat. How? Because "tired" describes a state of a living thing, and cats are alive, mats aren't. You didn't read word by word — your eyes jumped back and picked out the most relevant word.

A model needs to do the same thing. When deciding what word comes after "it," it must look back at all previous tokens and decide which ones are worth paying attention to and which ones to ignore.

That's what attention does.

The Intuition: Selective Reading

Imagine you're reading a long contract and encounter the pronoun "said party." You don't reread the whole thing from scratch — your eyes scan backward, find the most likely referent, and skip everything irrelevant.

Attention works almost identically:

• For each token, the model asks: "Which other tokens in the sequence are most relevant to what I need right now?"
• It assigns each token a relevance score (a weight between 0 and 1)
• High-weight tokens get close attention; low-weight tokens are nearly ignored
• Finally, it takes a weighted sum of all tokens' information to produce the "understanding" at this position

This isn't a novel idea — you do it every day. Attention just turns "selective reading" into a mathematical operation that can be computed.

The Mechanism: Query, Key, Value

Intuition in place — but how does the model actually compute "which tokens are relevant"? The answer is three vectors: Query (Q), Key (K), and Value (V).

The Library Analogy

Imagine a library:

• Query (Q): The question in your head as you walk in — "I need books about cat fatigue"
• Key (K): The label on each book's spine — "Animal Behavior," "Floor Materials," "Sleep Science"
• Value (V): The actual content inside each book

You compare your Query against each book's Key. The better the match (higher relevance), the more you want to open that book and read its Value.

In a Transformer, every token plays all three roles simultaneously:

1. When it's the "token being processed," it emits a Query — "What am I looking for?"
2. When it's a "candidate being looked at," it offers a Key — "What can I tell you?"
3. If it's selected, it contributes its Value — "Here's my actual content"

Computing Relevance: The QK Dot Product

Q and K are both vectors (derived from the embedding vectors from the previous article, passed through linear transformations). The more similar two vectors are, the larger their dot product.

relevance("it", "cat")  = Q_it · K_cat  = 8.2  ← high!
relevance("it", "mat")  = Q_it · K_mat  = 1.1  ← low
relevance("it", "the")  = Q_it · K_the  = 0.3  ← nearly ignored

Pass all scores through softmax (so they sum to 1) to get attention weights:

weights: [cat: 0.85, mat: 0.08, the: 0.02, sat: 0.03, on: 0.01, because: 0.01]

Finally, take the weighted sum of all tokens' Values:

output_it = 0.85 × V_cat + 0.08 × V_mat + 0.02 × V_the + ...

The result: the output representation for "it" is almost entirely the semantics of "cat." This is how the model "understands" that "it" refers to "cat."

The Attention Matrix: A Full Picture

If the sentence has N tokens, every token must compute relevance against all N tokens. This forms an N×N matrix called the attention matrix.

For "The cat sat on the mat" (simplified):

         The   cat   sat    on   the   mat
The     [0.1   0.2   0.1   0.1  0.3   0.2]
cat     [0.1   0.3   0.2   0.0  0.1   0.3]
sat     [0.0   0.5   0.2   0.1  0.0   0.2]
on      [0.1   0.1   0.2   0.1  0.1   0.4]
the     [0.2   0.1   0.1   0.1  0.2   0.3]
mat     [0.1   0.1   0.1   0.3  0.2   0.2]

Each row sums to 1. Read each row as: "how much attention does this token allocate to every other token." The sat row gives 0.5 to cat — because the most relevant answer to "who's sitting?" is the subject.

Multi-Head Attention: Multiple Reading Strategies

A single set of Q/K/V can only capture one type of relationship. But language has many layers:

• Syntactic: "sat" → subject is "cat"
• Coreference: "it" → refers to "cat"
• Modification: "tired" → modifies "it" (indirectly "cat")

Multi-head attention runs multiple sets of Q/K/V in parallel, each learning to focus on different types of patterns.

Imagine sending 8 people (8 heads) into the same library, each with a different task:

• Person 1 looks for subject-verb relationships
• Person 2 looks for pronoun references
• Person 3 looks for adjective-noun modification
• …

Each searches independently; their results are concatenated. The model gains multiple "reading perspectives" at once.

Positional Encoding: How the Model Knows Word Order

There's a problem we've been ignoring: the attention computation doesn't care about position at all. To attention, "The cat sat on the mat" and "mat the on sat cat The" are identical — it only looks at content, not order.

But word order obviously matters. "Dog bites man" and "man bites dog" mean very different things.

The fix is adding positional information (positional encoding) to the embeddings. The simplest way to think about it: each token gets a "position vector" added to its semantic embedding, telling the model which position it occupies.

final_input = embedding("cat") + position(2)

Now when attention computes the QK dot product, it considers both "what is this token" and "where is it."

Connecting Back: Why Attention Matters — and Where It Falls Short

Now you understand the core of how Transformers work:

1. Each token uses Q/K/V to compute relevance against every other token
2. A weighted sum produces a context-aware representation
3. Multiple heads capture different types of relationships in parallel
4. Positional encoding preserves word order

This explains why modern language models are so powerful — they can see any position in the entire input sequence in a single computation, unlike older architectures (RNNs) that processed word by word and gradually forgot earlier information.

But attention comes with costs:

• Computation scales quadratically with sequence length. N tokens require N×N comparisons. This is why context windows have limits — it's not that the model "can't remember," it's that very long inputs are too expensive (or too slow) to compute.
• Not every token gets proper attention. Even within the context window, in very long documents the attention weights for middle paragraphs tend to get diluted. This is why some models "miss details in the middle" of long inputs.

In the next article, we'll see where a model's capabilities come from — how three training phases (pretraining, SFT, and RLHF) transform a word-completion machine into a useful, safe assistant.

Going Deeper

• Stanford CS224N Lecture 5: Self-Attention and Transformers — motivates attention from a linguistics perspective
• Stanford CS336 Lecture 3: Transformers — engineering-focused with full implementation details
• Attention Is All You Need (Vaswani et al., 2017) — the original paper that launched the Transformer era
• The Illustrated Transformer (Jay Alammar) — perhaps the best visual explanation of Transformers on the internet

References

• Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS 2017.
• Stanford CS224N: Natural Language Processing with Deep Learning — Lecture 5: Self-Attention and Transformers.
• Stanford CS336: Language Modeling from Scratch — Lecture 3: Transformers.
• Alammar, J. (2018). The Illustrated Transformer.
• Su, J. et al. (2024). RoFormer: Enhanced Transformer with Rotary Position Embedding. Neurocomputing, 568, 127063.