The Transformer

Lesson 3 built a network from stacked matrix multiplications and non-linear activations. Every layer in that lesson used a fixed weight matrix, the same weights applied to every input, every time. The transformer changes this in one specific way: for one class of layer, the weights that decide how much one token influences another are computed dynamically, from the tokens themselves, every time the model runs.

Before the transformer, a model absorbed a sequence one token at a time, and had to compress everything it had seen so far into a single fixed-size memory. The transformer lets every token look directly at every other token, all at once, with no compression step in between.

This lesson works through exactly why the older, recurrent approach hit a wall, with the actual measured numbers behind that failure, derives the full self-attention calculation with a real worked matrix example, covers multi-head attention, compares two different ways of injecting position into a position-blind mechanism, and explains why the field converged on decoder-only architectures for generation.

Section 01Why recurrent networks hit a wall

For over two decades before the transformer, state-of-the-art sequence modeling relied on recurrent neural networks (RNNs) and their gated successor, Long Short-Term Memory networks (LSTMs). A recurrent network processes a sequence one step at a time, carrying a hidden state forward:

The hidden state at step t, hₜ, depends on the hidden state at the previous step, hₜ₋₁, plus the current input xₜ. LSTMs added memory cells and gating mechanisms, specifically input, forget, and output gates, to help preserve information across time steps. That design still left two structural limitations baked into how recurrence works.

The sequential bottleneck

Because hₜ strictly depends on hₜ₋₁, computing step t cannot begin until every step before it has finished. Training an LSTM over a sequence of length N requires N sequential operations, O(N). This directly fights against how modern accelerators, GPUs and TPUs, get their speed: massive parallelization across thousands of execution cores at once. A strictly sequential computation leaves that hardware fundamentally underused, because there is no way to compute step 5 before step 4 is done, no matter how many idle cores are sitting nearby.

Vanishing gradients and the fixed-size bottleneck

During backpropagation through time, the gradient computed at step N has to flow backward through N recurrent steps, being repeatedly multiplied by the recurrent weight matrix at every step:

Repeated multiplication like this either shrinks the gradient toward zero (vanishing) or blows it up (exploding) as the number of steps grows, the same phenomenon covered generally in Lesson 3. Even with LSTM gating designed specifically to let error flow through the cell state without decaying, early encoder-decoder translation models without any attention mechanism had a second, separate problem: the entire variable-length source sentence got compressed into one single fixed-size vector, before the decoder ever started generating output. Cho et al. (2014), in “On the Properties of Neural Machine Translation: Encoder-Decoder Approaches,” found that translation quality dropped sharply once source sentences grew past 10 to 20 words, a direct consequence of squeezing an arbitrarily long sentence through a fixed-size bottleneck. Bahdanau et al. (2014) partially fixed this with a dynamic alignment mechanism, an early ancestor of attention, but the fixed-size hidden state inside the recurrence itself remained a real limitation.

Measuring exactly how much context an LSTM actually uses

Khandelwal et al. (2018), in “Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context,” measured this directly. By systematically perturbing, shuffling, and dropping prior context tokens at test time on the Penn Treebank and WikiText-2 corpora, they quantified exactly how much of an LSTM’s supposed context window was actually being used.

• Effective context limit: LSTMs used only about 150 to 200 tokens of prior context, on average. Anything supplied beyond 200 tokens back produced no measurable improvement in perplexity.
• Local vs. distant history: LSTMs drew a sharp functional line between local context, the most recent 50 tokens, and everything further back.
• Word-order collapse: Models stayed sensitive to the exact word order within roughly the most recent 20 tokens, about one sentence. Past 50 tokens back, shuffling the order of prior tokens caused almost no change in perplexity at all.

That last finding is the sharpest one. Beyond 50 tokens back, an LSTM was not tracking syntax or relationships between specific words anymore. It had degraded distant context into an unstructured, order-agnostic “bag of topics,” useful for general subject matter but useless for anything requiring precise long-range grammatical or logical dependency.

Section 02What self-attention operates over

Self-attention eliminates recurrence entirely, replacing the O(N) sequential path with direct, O(1) pairwise connections between every pair of positions in a sequence, regardless of how far apart they are.

Attention works on continuous vectors, not raw text. Assuming tokenization has already happened, covered in the next lesson, each token index is mapped to a dense d-dimensional embedding vector. Because the self-attention calculation itself has no built-in notion of order, a positional encoding vector must be added directly to each token embedding, covered in Section 5, before attention runs:

Stacking all N of these combined vectors gives the input matrix X, holding both the semantic content and the position of every token in the sequence. Self-attention’s job is to transform these static, context-free vectors into dynamic, context-aware ones.

Section 03Query, Key, Value: the routing abstraction

To let information move dynamically between tokens, the model projects the input matrix X into three separate learned vector spaces, using three separate learned weight matrices, W_Q, W_K, and W_V.

• Query (Q): what a token is looking for. A verb might be searching for its direct object.
• Key (K): what a token offers, or what role it can play if selected. A noun phrase advertises its grammatical role this way.
• Value (V): the actual content a token contributes, once a Query and a Key are found to match.

Section 04The scaled dot-product attention formula

Vaswani et al. (2017), in “Attention Is All You Need,” define the full self-attention calculation as:

Four distinct stages happen inside this one formula.

1. Similarity scoring (QKᵀ). The dot product between every Query vector and every Key vector produces a raw compatibility score for every pair of tokens in the sequence.
2. Scaling (÷ √dₖ). The raw scores are divided by the square root of dₖ, the dimensionality of the key vectors. For large dₖ, dot products grow large in magnitude, which pushes softmax into a region where its gradient is nearly flat. Dividing by √dₖ keeps the variance of the dot products near 1, which keeps the gradient healthy during training.
3. Probability normalization (softmax). The scaled scores are passed through softmax, row by row, converting each row of raw scores into non-negative weights that sum to exactly 1 across that row. The resulting matrix A holds, at entry Aᵢⱼ, exactly how much attention token i pays to token j.
4. Contextual aggregation (× V). The attention weights scale the Value vectors. Every output token becomes a weighted sum of every Value vector in the sequence, weighted by exactly the probabilities computed in the previous step.

Section 05A fully worked matrix example

Here is the entire calculation, every number shown, for a 3-token sequence: “The”, “cat”, “sat”, using a representation dimension of d_model = dₖ = d_v = 2.

Setup: input and projection matrices

The combined input matrix (embeddings plus position, already summed):

X = [[1.0, 0.0], [0.0, 2.0], [1.0, 1.0]] (row 1 is “The”, row 2 is “cat”, row 3 is “sat”)

The learned projection matrices:

Step 1: project into Q, K, V

Step 2: raw attention scores, S = QKᵀ

Step 3: scale by √dₖ

Since dₖ = 2, the scaling factor is √2 ≈ 1.41421. Dividing every entry of S by √2:

Step 4: row-wise softmax

Using softmax(xᵢ) = eˣⁱ / Σⱼeˣʲ, applied independently to each row:

Row 1 (“The”): exponentials are [e^0.7071, e^0.0, e^0.7071] = [2.0281, 1.0000, 2.0281], summing to 5.0562. Dividing each by the sum gives Row₁ = [0.4011, 0.1978, 0.4011].

Row 2 (“cat”): exponentials are [e^1.4142, e^2.8284, e^2.8284] = [4.1132, 16.9182, 16.9182], summing to 37.9496. Dividing gives Row₂ = [0.1084, 0.4458, 0.4458].

Row 3 (“sat”): exponentials are [e^1.4142, e^1.4142, e^2.1213] = [4.1132, 4.1132, 8.3420], summing to 16.5684. Dividing gives Row₃ = [0.2483, 0.2483, 0.5035].

The full normalized attention matrix:

Step 5: output, O = AV

Reading the result

Token 2 (“cat”) splits its attention nearly evenly between itself and token 3 (“sat”), 0.4458 each, while assigning almost nothing, 0.1084, to token 1 (“The”). Its updated output vector, [1.1084, 1.3374], blends the noun’s own identity with the action performed on or by it. Token 3 (“sat”) assigns its single highest weight, 0.5035, to itself, and splits the rest evenly, 0.2483 each, between “The” and “cat”. Every output row is a genuinely new vector, computed as a weighted blend of everything in the sequence, not a lookup from a fixed table. That is the entire mechanism: four matrix operations, producing a context-aware representation for every token, all at once, with no step forced to wait for any other step to finish first.

Section 06Multi-head attention: several relationships at once

A single attention calculation, run once, tends to average together several different kinds of relationships into one blended signal, which dilutes specific syntactic and semantic information. Vaswani et al. (2017) fixed this with Multi-Head Attention (MHA): instead of computing attention once using the full d_model-dimensional vectors, MHA projects Q, K, and V h separate times, into lower dimensions, using h separate sets of learned weight matrices, and runs the same attention calculation independently inside each of those h subspaces.

Here Wᵢ^Q, Wᵢ^K, Wᵢ^V ∈ ℝ^(d_model × dₖ) are the per-head projection matrices, and Wᴼ ∈ ℝ^(hdᵥ × d_model) is a final learned matrix that recombines all h heads’ outputs back into a single d_model-dimensional vector. Because each head operates in its own lower-dimensional subspace, different heads are free to specialize in different kinds of relationships, all computed in parallel, then merged at the end.

What specific heads actually learn

Interpretability research, including Clark et al. (2019), “What Does BERT Look At? An Analysis of BERT’s Attention,” and visualization tools such as BertViz, found that individual attention heads in a trained model tend to specialize in identifiable, consistent linguistic roles.

Section 07Injecting position: two different solutions to the same problem

Self-attention performs a symmetric dot product between every pair of tokens. Swap the positions of two input tokens and the set of pairwise dot products stays exactly the same; only which output ends up assigned to which position changes. Self-attention is, by itself, permutation-invariant: it has no built-in sense of order at all. Position has to be injected deliberately.

Sinusoidal positional encoding (the original transformer)

Vaswani et al. (2017) used a fixed, non-learnable set of sine and cosine functions, one pair per pair of embedding dimensions:

pos is the token’s position in the sequence, and i indexes which pair of embedding dimensions the formula is generating. The authors chose this specific construction because, for any fixed offset k, the encoding at position pos+k can be expressed as a linear function of the encoding at position pos, using standard trigonometric angle-addition identities. This was meant to make relative position easy for the model to recover, even though the encoding itself is added as an absolute, fixed vector per position.

Rotary Position Embedding (RoPE)

Modern large language models, including LLaMA, Mistral, and PaLM, use a different approach: Rotary Position Embedding (RoPE), from Su et al. (2021), “RoFormer: Enhanced Transformer with Rotary Position Embedding.” Instead of adding a fixed vector to the input embedding, RoPE multiplies the Query and Key vectors by a rotation matrix, encoding position directly into the attention calculation itself rather than into the input.

For a 2-dimensional vector x = [x₁, x₂]ᵀ at position m, RoPE applies a rotation by angle mθ:

In a full d-dimensional space, RoPE splits the vector into d/2 separate 2-dimensional chunks and rotates each pair by its own position-dependent frequency. The key mathematical property is what happens when a rotated Query at position m and a rotated Key at position n are combined in the attention dot product:

The rotation matrices combine such that the result depends only on the relative offset, n − m, not on the absolute positions m and n themselves. This gives RoPE three concrete advantages: the attention dot product depends explicitly on relative distance, matching how natural language relationships actually work; the inner product naturally decays as relative distance |m − n| grows, automatically biasing the model toward nearby context; and RoPE extrapolates to sequence lengths beyond what the model was trained on more gracefully than a fixed table of absolute position vectors, which simply runs out of entries past its trained maximum.

Section 08Encoder-decoder vs. decoder-only

The original 2017 transformer paper introduced an encoder-decoder architecture, built for sequence-to-sequence translation. The encoder stack processes the entire input sequence at once, using unmasked, fully bidirectional self-attention, building a dense representation of the whole input. The decoder stack generates output one token at a time, using masked, causal self-attention over what it has generated so far, plus a separate cross-attention module that reads the encoder’s output.

Causal masking

Generating token i must depend only on tokens that came before it, never on tokens that come after. To enforce this without giving up parallel computation during training, models use causal masking. Before the softmax step of attention, an upper-triangular mask matrix M is added to the raw score matrix:

Since e^−∞ = 0, adding −∞ to every future position’s score before softmax forces that position’s attention weight to exactly zero after the softmax normalization. Every token still gets computed in parallel across the whole sequence during training, but the mask guarantees that no token’s output can be influenced, even slightly, by a token that comes after it.

Why decoder-only won for generation

BERT-style encoder-only models excel at bidirectional understanding tasks, and T5-style encoder-decoder models still work well for structured translation. But decoder-only models, such as GPT-4, LLaMA, Mistral, and Claude, became the dominant architecture for general-purpose language generation, for three concrete reasons.

1. A single, unified training objective. Decoder-only models reduce all of language modeling to one task: predict the next token, given everything so far. No separate encoder objective, no cross-attention machinery to train alongside it.
2. KV-cache efficiency. Once a token’s Key and Value vectors are computed, they never change on later generation steps, since causal masking guarantees nothing that comes after can affect them. A decoder-only model can cache every past token’s K and V tensors, and only compute a new Query vector for each new token generated, reducing the compute needed per generated token from O(N²) down to O(N).
3. Unified prompt and generation handling. Processing an input prompt and generating a response both happen under the exact same causal model, with no architectural seam between “reading the question” and “writing the answer.” This is a large part of why zero-shot prompting and in-context learning, covered in Lesson 9, work as cleanly as they do: there is only one mechanism running, start to finish.

Conclusion

Every limitation covered in Section 1, the sequential bottleneck, the vanishing gradient over long chains, the fixed-size compression of context, traces back to one design choice: recurrence forces information to travel step by step. Self-attention removes that constraint entirely by giving every token a direct, one-step path to every other token, computed through nothing more than the four-stage QKᵀ, scale, softmax, weighted-sum calculation worked through by hand in Section 5. Multi-head attention lets that mechanism specialize into several relationships at once instead of blurring them together. Positional encoding, whether the original sinusoidal scheme or the relative-distance-aware RoPE now standard in most modern models, solves the one problem attention does not solve on its own: knowing where each token sits. And the shift from encoder-decoder to decoder-only architectures was not a simplification for its own sake. It came from a concrete, measurable efficiency gain in exactly how key-value caching turns causal masking into cheaper inference.

The next lesson looks at what actually enters this whole calculation in the first place: how raw text gets broken into the tokens this lesson assumed were already sitting in matrix X.