Lesson 9 introduced chain-of-thought prompting, showed PaLM 540B’s jump from 17.9% to 58.1% accuracy on GSM8K, and covered Kojima et al.’s zero-shot “let’s think step by step” trigger. This lesson does not repeat that ground. It opens up the actual probability mechanics behind why writing out intermediate steps changes a model’s output at all, not just that it does, and previews what happens when that same reasoning gets connected to a real, checkable environment instead of staying entirely inside the model’s own head.
A model does not get smarter when you ask it to show its work. It gets more forward passes, more tokens to condition on, before it has to commit to an answer. That is the entire mechanism, expressed exactly.
This lesson derives the formal probability expansion that chain-of-thought prompting performs on a model’s output distribution, and closes with a preview of what happens once a reasoning chain gets connected to real tool calls and real feedback, a full-depth topic covered later in this course.
Section 01What chain-of-thought does to the probability distribution, exactly
Lesson 9 already covered chain-of-thought (CoT) prompting’s measured results. This section covers the part that lesson did not: what CoT actually does to the model’s underlying probability calculation, in exact mathematical form.
Standard few-shot prompting maps an input exemplar directly to a final output token, without anything in between. Written as a conditional probability, the model computes P(y | x), the probability of output y given input x, in essentially one shot. On tasks that require genuine multi-step arithmetic or logical reasoning, this frequently fails, not because the model lacks the underlying knowledge, but because it never gets to allocate any intermediate computation toward actually working the problem out. It has to predict the final answer token directly, with nothing generated in between to condition on.
Chain-of-thought prompting changes the object the model is computing. Instead of mapping directly from input to output, each in-context demonstration now inserts an intermediate natural-language reasoning path R between the input and the final output. Given a test input xtest, and a full CoT-formatted context CCoT built from L worked reasoning steps and M answer-formatted exemplars, the joint probability the model is actually computing becomes:
R = (r₁, r₂, ..., rL) is the sequence of reasoning tokens the model generates before it ever produces a final answer token. Read this formula the same way Lesson 6 unpacked the autoregressive chain rule for plain next-token prediction: the first product term generates the reasoning tokens one at a time, each one conditioned on everything generated before it, and only once that entire reasoning sequence R exists does the second product term generate the final answer tokens, now conditioned not just on the original input but on the full reasoning path R that came before it.
This is the precise mechanism behind why chain-of-thought works at all. Because the model is autoregressive, generating one token at a time and attending back over every token generated so far, as covered in Lesson 4, inserting R between the input and the answer literally expands how much computation happens before the model has to commit to a final answer. Every token in R is itself a full forward pass through the entire network, and every one of those forward passes gets to attend over every previous reasoning token. The model’s computational footprint scales directly with how long the reasoning sequence R is allowed to run. This is not a rhetorical framing device for a human reader. It is additional compute, structurally built into the generation process itself.
Why this explains the emergent-ability threshold from Lesson 9
Lesson 9 reported that CoT prompting only reliably helps above roughly 100 billion parameters, and can actively hurt smaller models. The probability expansion above explains why that threshold exists at all, rather than CoT simply helping a little more as scale increases. Every term inside both product expressions in P(R, Y | xtest) is still just next-token prediction, exactly as covered in Lesson 6. Extra reasoning tokens only help if the model generating them is capable enough to make each rⱼ a logically sound continuation of what came before. A model below the emergent-ability threshold generates reasoning tokens that are fluent, grammatically well-formed continuations of the sequence so far, exactly what next-token prediction optimizes for, but that are not necessarily logically valid steps in an actual derivation. Once that flawed rⱼ sits inside the context that P(yᵢ | y<i, R, xtest, CCoT) conditions on, the final answer terms are now conditioning on a corrupted premise, which actively drags accuracy down rather than helping it. Above the threshold, the reasoning tokens the model generates are reliable enough, as a byproduct of the model’s scale, that R functions as genuine intermediate computation instead of injected noise.
A worked illustration of the expansion
Take the exact tennis-ball example from Lesson 9: “Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?”
Under standard prompting, the model computes one term: P(y | xtest, Cstandard), where y is directly the token “11.” That single probability has to be high enough, on its own, for the model to output the correct number. There is no intermediate structure the model can lean on if its first, direct attempt at the arithmetic is wrong.
Under chain-of-thought prompting, the model instead computes a chain of much easier, smaller probabilities, one token at a time: P(r₁ = “Roger” | ...), P(r₂ = “started” | ...), continuing on through a reasoning path like “Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls.” Each of those individual next-token predictions is a far easier prediction than jumping straight to “11,” because each one only has to continue a short, locally coherent arithmetic statement, not solve the entire problem in a single step. Once R = “Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11.” exists in the context, the final term, P(y = “11” | y<i, R, xtest, CCoT), is no longer a difficult, high-uncertainty prediction. It is close to reading the answer directly off of a correct arithmetic statement that the model itself just generated, one easy token at a time.
This is why the accuracy gap in Lesson 9’s GSM8K numbers is as large as it is. Standard prompting asks a single, hard probability term to carry the entire burden of correctness. Chain-of-thought prompting spreads that same burden across many easier, smaller probability terms, each of which is individually far more likely to be predicted correctly, at the cost of generating more tokens before an answer arrives.
The direct cost: more tokens, more latency, more spend
The expansion in Section 1 is not free. Every reasoning token in R is an additional forward pass through the entire network, and Lesson 5 covered how commercial APIs bill directly by the token. A chain-of-thought response that runs 150 reasoning tokens before its final answer costs roughly 150 tokens more, in both latency and API spend, than a standard prompt that jumps directly to a one-token answer, since autoregressive generation produces exactly one token per forward pass. This is the direct, mechanical tradeoff behind the pattern-selection table in Lesson 27: a technique that reliably turns a hard, single-shot prediction into several easy ones is worth its added token cost exactly when the task genuinely needs that extra computation, and is pure waste when it does not.
Section 02From a private reasoning chain to a checked one: a preview
Everything covered in this lesson and in Lesson 9 so far happens entirely inside the model’s own generated text. The reasoning sequence R is a sequence of tokens the model produces from its own training, with nothing external ever checking whether any individual step in R is actually true. If the model’s training data or its own internal computation leads it toward a wrong intermediate step, nothing in the mechanism described above catches that error. The model simply continues generating, fluently, from a premise that happens to be false.
A natural next question is what happens once a reasoning chain is no longer purely private: what if the model could pause partway through R, take an action against a real external system, a search query, a calculator, a database lookup, and read back a real, checked result before continuing to reason? That single change, interleaving generated reasoning with real, executed actions and real observations, is the foundation of a family of techniques covered at full depth in Lesson 27, starting with the ReAct pattern. Lesson 27 also covers Tree of Thoughts, which addresses a different, specific gap: everything in this lesson’s probability expansion still produces one single, linear sequence R. There is no way, structurally, for the model to try one reasoning path, discover partway through that it was a dead end, and back out to try a different one. That capability, along with what happens when an agent can learn from a previous failed attempt without any weight update at all, is covered once the vocabulary for tool use and agents from Lesson 17 is in place.
Conclusion
Chain-of-thought prompting is not a trick that makes a model reason differently. It is a structural change to what probability the model is computing: instead of mapping an input directly to an output in one step, it interposes a generated reasoning sequence R, and every token of R is a genuine additional forward pass the model gets to use before committing to a final answer. That mechanism is exactly why the technique’s benefit depends so sharply on model scale, covered with real benchmark numbers in Lesson 9: a capable enough model turns R into real computation, while a less capable one turns it into fluent noise that corrupts its own final answer. Everything in this lesson still happens inside one linear, private, ungrounded sequence of generated tokens. The next lesson turns to a different limit on that sequence entirely: how large it can actually get, and why a model’s context window is not the same thing as memory.