Context Windows

Lesson 4 showed that self-attention computes a score between every pair of tokens in a sequence, and that this cost grows with the square of the sequence length. This lesson works out exactly what that means in practice: why a context window has a hard limit, why doubling it costs far more than double the compute, and why calling it “memory” is a category error that leads to real misunderstandings about what a model actually does between requests.

A context window is not a place a model stores things. It is the input size of one large, stateless calculation, repeated in full every single time, with nothing carried over from the last time it ran.

This lesson derives the exact compute and memory cost of attention as sequence length grows, explains precisely why “memory” is the wrong word for what a context window does, and closes with a dated snapshot of real context window sizes across frontier models as of this course’s sourcing.

Section 01The quadratic cost, derived exactly

For an input sequence X ∈ ℝ^L×d_model of length L, self-attention computes Query, Key, and Value projections, exactly as covered in Lesson 4:

The step that determines the entire cost profile of attention is the matrix multiplication that produces the raw similarity scores:

Every entry S_i,j, for i, j ∈ [1, L], is a dot-product score between token i’s Query and token j’s Key. Because this score gets computed for every possible pair of positions in the sequence, not just adjacent ones, the resulting matrix S has exactly L² entries.

Computational cost

Computing QKᵀ for a single attention head requires 2 · L² · d_k floating-point operations. Across h attention heads and l transformer layers, the total attention computation across the whole model scales as:

Sequence length L is squared in this expression. Every other factor, model depth, hidden dimension, number of heads, only ever multiplies the total linearly.

Memory cost

The problem is not just compute. The raw L × L score matrix S has to actually exist somewhere in memory before softmax can be applied to it, and that has to happen once per attention head per layer. Materializing l · h separate L × L matrices produces a spatial memory cost of:

Kernel-level optimizations such as FlashAttention avoid writing the full L × L matrix out to a GPU’s slower High Bandwidth Memory, instead tiling the computation directly inside fast on-chip SRAM. This is a real and valuable engineering optimization, and it meaningfully reduces the memory-bandwidth bottleneck in practice. But it does not change the underlying arithmetic. The number of floating-point multiplications the QKᵀ step requires is still fundamentally O(L²), no matter how cleverly the intermediate results are stored and moved around.

What doubling the context window actually costs

Because the cost scales with L², doubling the context window does not double the compute needed for attention. It roughly quadruples it. Going from a 100,000-token context to a 200,000-token context is not a 2x increase in attention compute; it is closer to a 4x increase, since (2L)² = 4L². This single fact is the direct, mechanical reason extremely long context windows are expensive to serve, and why research into sparse attention, linear attention, and state-space model alternatives remains an active area of work: none of them are free upgrades, they are all attempts to trade away some of this quadratic cost for a different set of tradeoffs.

Section 02Why “memory” is the wrong word

In ordinary software engineering, “memory” implies a few specific things: state that gets written once and can be read back later, values that persist and can be updated over time, and a system that remembers something between one operation and the next. A large language model’s context window has none of these properties during inference, and being precise about this matters, because sloppy language here leads directly to sloppy assumptions about what a deployed system actually does.

The weights are fixed; the context is not stored inside them. A model’s parameters, θ, the same parameter vector introduced in Lesson 6, are fixed constants sitting in GPU memory at inference time. Nothing about processing a prompt changes those weights. The tokens in the context window are temporary numerical input, fed into a fixed calculation, not new information being written into the model’s long-term parameters.

Nothing persists once a request ends. Once a generation pass finishes and the request completes, every intermediate Key and Value vector computed for that context, the same KV-cache tensors covered in Lesson 4, gets discarded. The model retains absolutely nothing from that specific prompt once the response is returned. There is no internal state left over that the next request could draw on.

“Conversation history” is re-sent, not remembered. A standard API is stateless by design. What looks like a model remembering earlier turns in a multi-turn conversation is, mechanically, the calling application resending the entire prior message sequence back into the model on every single new request. The appearance of continuity comes entirely from the application layer re-running the full attention calculation over the accumulated conversation buffer each time, not from the model holding onto anything internally between calls.

This distinction is not pedantic. It has direct consequences for anyone building on top of these models: a longer conversation does not make a model “know you better” in any persistent sense, it makes every subsequent request more expensive, because of the quadratic cost derived in Section 1, and every part of that conversation has to be resent, in full, every time.

Section 03A snapshot of context window sizes today

Specific context window numbers change quickly as providers release new models and API tiers. The table below is a dated snapshot, sourced from official provider documentation, and should be treated as a point-in-time reference, not a permanent fact.

(Compiled from official provider documentation as of this course’s sourcing. Treat every number in this table as subject to change; refresh before relying on it for production planning.)

Note the Llama 3 row specifically: a model does not have to be pretrained at its eventual maximum context length. Llama 3 was pretrained at an 8,192-token window and extended afterward using RoPE’s relative-position property, covered in Lesson 4, which extrapolates to unseen sequence lengths more gracefully than a fixed table of absolute position embeddings would.

Conclusion

Every property in this lesson traces back to the same O(L²) fact derived in Section 1. A context window has a real, hard ceiling because the attention mechanism that makes a transformer work at all gets quadratically more expensive as that window grows, in both compute and memory, and no amount of clever engineering around how the intermediate matrix gets stored changes the underlying arithmetic. And a context window is not memory in any sense a software engineer would normally use that word: the weights are fixed, nothing persists between requests, and what looks like conversational continuity is really the full history being resent and recomputed from scratch every single time. “Lost in the middle” and “context rot,” the specific ways model performance actually degrades as a context window fills up rather than just costing more, get their first mention here and their full treatment later in this course, in the coding-agent and harness-engineering material, once the full picture of agent harnesses and failure modes is in place.

The next lesson turns to a different kind of input entirely: how images, audio, and generated content get forced into the same sequence-of-vectors shape that text tokens already occupy.