Prompting Fundamentals

Most advice about prompting comes from habit, not from evidence. Some habits help, with real numbers behind them. Some do nothing at all, no matter how often they get repeated online.

This lesson separates the two groups. It explains what a prompt does inside the model, walks through the exact mechanics of in-context learning, shows measured results from primary research, and audits a set of popular prompting habits against controlled studies, including the well-known “take a deep breath” case, which turns out to be a cautionary tale rather than a tip worth copying.

A prompt does not train the model. It conditions which pattern, out of everything the model learned during training, it continues.

Section 01What a prompt actually does, precisely

A large language model computes a probability distribution over its next word-piece, given everything written so far, then samples from that distribution. A prompt is the text placed before that calculation begins.

Giving a model a prompt does not update its internal parameters. Those stay completely fixed at inference time. What changes is the input sequence itself, which reshapes the self-attention calculations across every layer of the network, which in turn reshapes the probability distribution over the next word-piece.

This is worth being precise about, because it is the exact mechanism behind in-context learning: a model’s ability to perform a brand-new task, described only in the prompt, without any weight update at all. The original GPT-3 paper (Brown et al., 2020) formally separated model evaluation into three settings, based on how many worked examples sit in the prompt before the real question:

• Zero-shot: the prompt contains only a natural-language instruction and the target input. No worked examples.
• One-shot: exactly one worked example, an input paired with its correct output, appears before the target input.
• Few-shot: several worked examples, commonly somewhere between 10 and 100 in the original GPT-3 experiments, appear before the target input, bounded by how much fits in the model’s context window (2,048 tokens, for the original GPT-3).

None of these settings touch the model’s weights. The worked examples work entirely by conditioning the attention patterns across the prompt: they narrow down, through pure pattern demonstration, which output format and which kind of reasoning the model should continue with.

A worked comparison

``text Classify the sentiment of the following movie review as Positive or Negative. Review: “The pacing was uneven and the dialogue felt forced.” Sentiment: ``

```text Classify the sentiment of the following movie review as Positive or Negative.

Review: “An absolute masterpiece of cinematography and acting.” Sentiment: Positive

Review: “Boring, predictable, and far too long.” Sentiment: Negative

Review: “The pacing was uneven and the dialogue felt forced.” Sentiment: ```

The examples in the second prompt do two separate jobs at once. They demonstrate the task, and they demonstrate the exact output format to copy: one word, directly after “Sentiment:”, with no explanation, no preamble, and no hedging.

Does adding examples actually help? It depends heavily on the task.

Brown et al. (2020) measured this directly, across several benchmarks, using the full 175-billion-parameter GPT-3.

Two patterns stand out. On tasks that reward recalling and formatting a fact correctly (TriviaQA, Natural Questions), few-shot examples produced a real, substantial lift: 6.9 to 15.3 percentage points. But on the ARC Challenge, a benchmark that needs multi-step scientific reasoning, few-shot examples added almost nothing over zero-shot, moving accuracy by just 0.1 points. Giving a model more examples is not a universal fix. It works by demonstrating a pattern, and pattern demonstration only helps when matching a pattern is most of what the task requires.

Section 02Chain-of-thought: giving the model room to compute

For genuine multi-step reasoning tasks, a different technique works far better than adding more examples: getting the model to write out intermediate reasoning steps before committing to a final answer. This is chain-of-thought (CoT) prompting, from Wei et al. (2022).

``text Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now? A: The answer is 11. ``

``text Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now? A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11. ``

Why does this actually change the outcome, and not just the appearance of the outcome? Standard prompting maps an input directly onto a final output token. The model has only as much internal computation as fits in the layers processing that single output position. Complex arithmetic or multi-step logic frequently exceeds what that gives it, and the model produces a fluent but wrong final number. Chain-of-thought prompting inserts an explicit reasoning sequence between the question and the answer. Because the model generates one token at a time, and each new token can attend back over every token generated so far, writing out the intermediate steps literally gives the model more computation, spread across more forward passes, before it has to commit to a final answer. The reasoning tokens are not just for a human reader. They are additional compute the model gets to use.

The effect is large, but only above a size threshold

On the GSM8K grade-school math benchmark, PaLM at 540 billion parameters jumped from 17.9% accuracy under standard 8-shot prompting to 58.1% accuracy under 8-shot chain-of-thought prompting, a 40.2-point gain. GPT-3 at roughly 175 billion parameters (the code-davinci-002 checkpoint) showed a nearly identical jump: 19.6% to 60.1%, a 40.5-point gain.

Wei et al. describe chain-of-thought reasoning as an emergent ability: a capability that is essentially absent below a certain model scale, and appears fairly suddenly above it, roughly around 100 billion parameters in their experiments. Below that threshold, chain-of-thought prompting can actually make results worse. Smaller models generate reasoning steps that read as fluent, grammatical text, but are not logically sound, and those flawed intermediate steps actively corrupt the final answer, rather than helping produce it. If you are working with a small model, do not assume chain-of-thought prompting will help. Test it directly, because the research says it can go the other way.

You do not need hand-written examples to get most of this effect

Kojima et al. (2022) found a two-stage trick that gets much of the chain-of-thought benefit without any worked examples at all.

Stage 1, reasoning extraction: append a trigger phrase to the question.

``text Q: A juggler can juggle 16 balls. Half of the balls are golf balls, and half of the golf balls are blue. How many blue golf balls are there? Let’s think step by step. ``

The model generates its own reasoning: it works out that half of 16 is 8 golf balls, then half of 8 is 4 blue golf balls.

Stage 2, answer extraction: append the full reasoning from stage 1, then a second trigger phrase asking for a clean final answer.

``text [Original question] Let’s think step by step. [Model’s reasoning from stage 1] Therefore, the answer (arabic numeral) is ``

The model then outputs a single clean number: 4.

Using InstructGPT, this two-stage, zero-example pipeline raised accuracy on the MultiArith benchmark from 17.7% to 78.7%, and on GSM8K from 10.4% to 40.7%. Compare this against the full table:

Zero-shot CoT beats standard prompting by a wide margin, using nothing but one added sentence. It still falls short of few-shot CoT, because hand-written worked examples give the model an explicit structural template for its reasoning, which reduces drift and off-topic steps. But as a zero-cost, always-available first step, the “let’s think step by step” trigger is close to a free improvement.

Section 03API-level roles: system, user, and assistant

Production systems do not send one block of raw text to a model. They send a structured list of messages, each tagged with a role, through an API such as the Anthropic Messages API or the OpenAI Chat Completions API.

``json [ { “role”: “system”, “content”: “You are a database migration engine. Output valid SQL statements only.” }, { “role”: “user”, “content”: “Convert this PostgreSQL table schema to MySQL...” }, { “role”: “assistant”, “content”: “ALTER TABLE users ADD COLUMN” } ] ``

• System: Sets the standing constraints for the entire conversation: the model’s persona, its boundaries, its operating rules. This is placed first in the token sequence, and its instructions apply to every token generated afterward.
• User: Carries the specific, turn-by-turn query or task. Treat user content as untrusted input, because in a deployed application it usually is exactly that: text from an outside party, not from you.
• Assistant: Holds the model’s own previous replies in a multi-turn history, giving the conversation continuity.

Response prefilling: a structural trick worth knowing

Some APIs (Anthropic’s among them, for supported models) let you append an unclosed assistant turn at the end of the message array, effectively telling the model where to start generating from. If you supply a prefill such as:

``json {“role”: “assistant”, “content”: “{\n \”status\“: \”success\“,\n \”data\“:”} ``

the model skips any conversational preamble (“Sure, here is your JSON output:”) and generates directly from inside the already-open JSON object. This enforces output formatting, reduces variance in the response shape, and lowers the number of tokens generated before useful output begins.

Why keeping the system prompt stable saves real money

API messages get serialized into one continuous token sequence before the model ever sees them. Providers commonly cache the computed representation of a prefix of that sequence, keyed by its exact content. If you change the system prompt on every call, you invalidate that cached prefix every single time, forcing the full computation to be redone from the start of the sequence. Keeping the system prompt static, and putting everything that changes into the user turn, maximizes cache reuse, which measurably reduces both cost and the time before the first token appears.

Section 04Prompts are far more fragile than they look

Primary research shows that changes to a prompt that look cosmetic can cause large, unpredictable swings in accuracy, on the exact same task, with the exact same model.

Order sensitivity: same content, different order, wildly different results

Lu et al. (2021) tested this precisely. Using GPT-2 XL (1.5 billion parameters) on the SST-2 sentiment classification task, they took a fixed set of 4 few-shot examples and tried every one of the 24 possible orderings, without changing a single word of content inside any example.

• Best ordering: 88.7% accuracy
• Worst ordering: 51.6% accuracy
• Swing: 37.1 percentage points, from order alone

Worse, the researchers found that a good ordering on one model size gives no guarantee of being good on another. Measuring the statistical correlation between which orderings worked well on a 2.7-billion-parameter model versus a 175-billion-parameter model, they found only a weak, unreliable relationship. A prompt order tuned on a small development model carries no guarantee of transferring to a larger production model.

Three specific biases that drive this fragility

Zhao et al. (2021), in “Calibrate Before Use,” identified three systematic biases baked into how in-context learning actually works, which together explain much of this fragility:

1. Majority label bias. The model over-predicts whichever label appears most often among the few-shot examples in the prompt, regardless of what the actual input says.
2. Recency bias. The model leans heavily on whichever example sits closest to the real question at the end of the prompt, because tokens near the end of the sequence receive stronger positional weight in the attention calculation.
3. Common token bias. Autoregressive models carry a built-in bias toward words that were frequent in their training data (predicting “America” over “Albania,” for instance) independent of what the prompt actually says.

To measure how severe this is, Zhao et al. fed models a content-free input, such as the literal string “Input: N/A”, and looked at the output label probabilities. An unbiased classifier facing a meaningless input should split its guesses close to evenly across the possible labels. In practice, models showed sharply skewed probabilities on this meaningless input, for example assigning roughly 61% probability to one label and 39% to another, on a task that should have been a coin flip. That skew reveals a hidden prior baked into the model’s behavior, sitting underneath every real prediction it makes.

Zhao et al.’s fix, called contextual calibration, measures this bias on a content-free input first, then applies a correcting transformation to every real prediction to cancel it out. Calibrating this way measurably stabilized accuracy across different prompt templates and different exemplar orderings.

Section 05The “take a deep breath” case: a warning about viral prompting advice

One of the clearest cautionary tales in prompt engineering research came from Google DeepMind’s own paper on automated prompt optimization, “Large Language Models as Optimizers” (Yang et al., 2023), and it is worth understanding exactly what the paper did and did not claim.

The paper introduced OPRO (Optimization by PROmpting): an automated search loop where one “optimizer” model repeatedly proposes new candidate instruction phrasings, a separate “scorer” model gets evaluated using each candidate, and the score gets fed back into the optimizer to propose the next candidate. Running this loop against PaLM 2-L on the GSM8K training set, the search eventually landed on this phrase as its best discovered instruction:

“Take a deep breath and work on this problem step-by-step.”

That exact phrase scored 80.2% training accuracy, ahead of a simpler human-written instruction, “Let’s do the math!”, which scored 78.2%.

Online discussion turned this into a widely repeated claim: that telling a model to “take a deep breath” fundamentally improves its reasoning. Looking at what the paper actually demonstrated tells a narrower story:

1. The phrase was the output of an automated local search, run against one specific model (PaLM 2-L) on one specific dataset split (GSM8K). It was not proposed, or tested, as a general-purpose instruction.
2. Follow-up evaluations against other model families, including Llama-2, Claude 3, and GPT-4, and against other task types, found the phrase produced inconsistent results, often with no statistically meaningful difference from a plain instruction.
3. The phrase is best understood as a variant of the zero-shot chain-of-thought trigger covered earlier in this lesson (“step-by-step” is doing the real work). Any benefit it carries most likely comes from the same mechanism: more room for intermediate reasoning tokens, not from anything resembling psychological framing.

The lesson here is not “this phrase is useless.” It is that a result discovered by search, against one model, on one benchmark, is not evidence of a universal rule, and treating it as one is a common trap.

Section 06What has evidence, and what does not

If a prompting tip circulates online with no benchmark attached to it, treat it as an untested claim, not a rule. Test it against your own task before you build it into a production system.

Conclusion

A prompt reshapes which pattern the model continues. It does not teach the model anything permanent. Two techniques carry strong, repeated, quantified evidence: worked examples for tasks that reward matching a pattern, and explicit step-by-step reasoning for tasks that need real computation. Beyond those two, prompt behavior is measurably fragile, sensitive to exemplar order and to hidden biases baked into how in-context learning works, and much of what circulates as prompting wisdom, including some advice that traces back to real research, does not hold up once someone checks it against a controlled study on a different model.