Lesson 6 established that pretraining minimizes cross-entropy loss, and nothing in that objective directly rewards truth. This lesson takes that observation and makes it precise: a worked example showing gradient descent actively rewarding a false completion over a true one, why a model’s own reported confidence cannot be trusted as a truth signal, and which mitigation techniques actually move the needle, measured against a formal benchmark.
A model that hallucinates is not malfunctioning. It is doing exactly what its training objective asked it to do: predict the statistically likely next token. Truth was never part of that calculation, and treating fluency as a proxy for truth is the single most common mistake in deploying these systems.
This lesson formally separates two distinct kinds of hallucination, walks through the exact arithmetic of why maximum-likelihood training can penalize a true completion and reward a false one, explains three separate reasons a model’s own log-probabilities cannot be trusted as a confidence signal, covers TruthfulQA’s benchmark structure and baseline numbers, and audits mitigation techniques against a table of what is measured and what is folklore.
Section 01Two different failure modes: intrinsic vs. extrinsic hallucination
In casual usage, “hallucination” covers almost any generation failure. Ji et al. (2023), in a foundational survey on natural language generation, define it more precisely, splitting hallucination into two structurally different categories based on the model’s relationship to its own input context.
Intrinsic hallucination is generated content that directly contradicts information already present in the prompt or reference context. If a prompt contains a reference document stating “The first Ebola vaccine was approved by the FDA in 2019,” and the model outputs “The FDA approved the first Ebola vaccine in 2021,” that is intrinsic: the output violates information the model was explicitly given.
Extrinsic hallucination is generated content that introduces information not present in, and not derivable from, the source context at all. If a prompt states only “Jane Doe is a computer scientist,” and the model adds “Jane Doe received her PhD from MIT in 2012,” that addition is extrinsic. The added detail might happen to be true in the real world, or it might not be, but either way it is ungrounded relative to what the model was actually given.
| Type | Relationship to source | Failure mode | Primary cause |
|---|---|---|---|
| Intrinsic | Direct contradiction | Output conflicts with the provided context | Context compression failures, attention misalignment, the model’s own parametric knowledge overriding what it was told |
| Extrinsic | Ungrounded expansion | Output introduces claims absent from and unverifiable by the prompt | Next-token completion bias, the model falling back on parametric memory instead of staying bounded by context |
This distinction matters because the two failures call for different fixes. Intrinsic hallucination signals that the model is failing to actually attend to and preserve the tokens already in its context window. Extrinsic hallucination signals the opposite problem: the model is reaching past its context entirely and drawing on whatever associations it picked up during pretraining.
Section 02Why maximum-likelihood training does not optimize for truth
Lesson 6 covered the cross-entropy pretraining loss in full. Restated here in the notation this lesson needs: given a sequence of tokens x = (x₁, x₂, ..., xT), model parameters θ are trained to minimize negative log-likelihood over a corpus D:
using the same autoregressive chain-rule factorization from Lesson 1 and Lesson 6:
This objective minimizes the KL divergence, covered in Lesson 1, between the model’s learned distribution and the statistical distribution of its training corpus. It rewards the model precisely for matching how often word sequences actually co-occurred in that corpus, with no separate mechanism anywhere in the math that checks whether any given sequence is factually true. Lin et al. (2021) documented a direct consequence of this: because an unaligned model is trained purely to imitate the statistics of human text, and human text is full of uncorrected misconceptions, superstitions, and folklore, minimizing perplexity on web text actively incentivizes reproducing those same false beliefs. Lin et al. call this phenomenon imitative falsehoods, and found something counterintuitive: under pure maximum-likelihood pretraining, larger base models can become less truthful, not more, because their greater capacity lets them learn and reproduce widely circulated misconceptions even more faithfully.
A worked example: gradient descent rewarding the false answer
Take a simplified vocabulary V = {“The”, “sun”, “revolves”, “around”, “the”, “Earth”, “Sun”} and a prompt prefix x<t = (“The”, “sun”, “revolves”, “around”, “the”).
Suppose an uncurated training corpus contains 100,000 instances of historical, mythological, or metaphorical text completing this prefix with “Earth” (as in “the sun revolves around the Earth,” reflecting a pre-Copernican or metaphorical framing), and only 20,000 instances of scientifically accurate text completing it with “Sun” (clarifying that the sun does not revolve around the Earth). The empirical distribution in the corpus is:
The cross-entropy loss for a single prediction is L = −log Pθ(xt | x<t). Consider what happens to that loss under each possible model prediction.
If the model predicts the factually correct completion, xt = “Sun”, with the model assigning it probability Pθ = 0.167 (matching the corpus’s own minority rate for that completion):
If the model predicts the factually incorrect completion, xt = “Earth”, with the model assigning it probability Pθ = 0.833:
The loss for predicting the true completion is roughly ten times larger than the loss for predicting the false one. Gradient descent, exactly as covered in Lesson 2, moves parameters in whichever direction reduces total loss. Given this training distribution, gradient descent will actively push the model toward assigning higher probability to the false completion and lower probability to the true one, because doing so is what minimizes cross-entropy loss across the corpus as it actually exists. This is not a bug in the optimizer. The model is being correctly and precisely optimized to imitate the statistics of its training distribution, and nothing in that objective distinguishes a popular falsehood from an accurate but less common fact.
Section 03Why a model’s own confidence cannot be trusted
A common assumption in production engineering is that a model’s own reported token probabilities, its logprobs, can serve as a reliability signal: if the model assigns a token high probability, surely it is more likely to be correct.
Formally, given logits zt ∈ ℝ^|V| produced by the model’s final projection layer, exactly the unembedding step from Lesson 6, the probability assigned to a specific vocabulary token v at temperature T is:
Sequence-level confidence for a full generated sequence S = (x₁, ..., xN) is typically computed as the mean log-probability across all N tokens:
This number quantifies conditional statistical likelihood under the model’s learned distribution. It does not measure the epistemic probability that the resulting claim is actually true, and three separate mechanisms cause that gap.
Miscalibration under free-form generation. Kadavath et al. (2022), in “Language Models (Mostly) Know What They Know,” found that while base models show moderate calibration on structured multiple-choice formats, that calibration collapses during free-form text generation. Models routinely assign extremely high probability (P > 0.99) to entirely fabricated entities, because autoregressive decoding produces sharply peaked distributions once a specific phrasing has already begun, regardless of whether that phrasing describes something real.
Exposure bias and error compounding. During training, the model conditions on ground-truth previous tokens, a technique called teacher forcing. During actual inference, it conditions on its own previously generated tokens, x̂_{<t}, which it had no such guarantee about. If an early token in a generated sequence happens to be a hallucination, produced with high confidence purely because of local phrasing, that token gets appended to the context and the model then conditions all subsequent generation on its own mistake. Each following token can carry a high logprob precisely because it is a logically consistent continuation of an already-false premise. This compounding effect is sometimes called snowballing: confidence keeps climbing even as the underlying claim drifts further from anything true.
Distortion from preference tuning. RLHF and DPO, the alignment techniques covered in Lesson 1, systematically reshape a model’s calibration curve. Optimizing against a human preference reward model introduces sycophancy, where the model produces false statements that simply agree with whatever the user’s prompt already assumed, and it rewards confident, fluent phrasing over honest hedging, because human annotators tend to penalize responses that sound hesitant or uncertain. The practical result is that post-RLHF models frequently output false claims with high logprobs, not despite alignment training, but partly because of it.
Section 04TruthfulQA: measuring the gap directly
To quantify how often models produce false statements, Lin et al. (2021) introduced TruthfulQA, a benchmark of 817 questions across 38 categories, including health, law, finance, politics, and climate, deliberately engineered to be adversarial: every question targets a topic where human-written text is known to contain widespread misconceptions.
TruthfulQA evaluates models on two tracks.
Generation track. The model produces a free-form answer, scored by classifiers fine-tuned on human judgments (GPT-judge and GPT-info). Two metrics come out of this: % True, the fraction of responses that are factually correct regardless of whether they say anything useful, and % True * Info, the fraction that are both correct and informative, which penalizes evasive non-answers like “I have no comment” that would otherwise inflate the raw % True score.
Multiple-choice track. MC1 measures whether the model assigns its single highest completion probability to the one correct answer among several options. MC2 measures the total normalized probability mass the model assigns across every correct option relative to every incorrect one.
Baseline results
| Model | Track | % True | % True * Info | MC1 |
|---|---|---|---|---|
| Human baseline | Generation | 94.0% | 87.0% | N/A |
| GPT-3-175B (helpful prompt) | Generation | 58.0% | 21.0% | N/A |
| GPT-3-175B (default prompt) | Generation | ~38.0% | ~21.0% | N/A |
| GPT-J-6B | Generation / MC | 43.0% | 22.0% | 20.1% |
| GPT-Neo-125M | Generation / MC | 60.0% | 14.0% | 25.6% |
| LLaMA-1-7B baseline | Generation / MC | ~32.5% | ~24.0% | 23.1% |
| LLaMA-2-7B baseline | Generation / MC | ~49.1% | ~43.2% | 39.5% |
| GPT-4 (anti-hallucination tuned) | System-card benchmark | ~60.0% | N/A | N/A |
(These numbers are a snapshot from the papers and system cards cited; specific model versions and their scores will change as newer models are evaluated.)
Two patterns in this table are worth reading carefully. GPT-3 175B, under a helpful prompting condition, produced false-but-informative statements 42% of the time (100% minus the 58% True rate). And smaller models sometimes achieved a deceptively high raw % True score purely by giving uninformative, hedging answers like “I don’t know,” which the separate % True * Info metric was specifically designed to catch. In the original benchmark, no unassisted model significantly outperformed random guessing on the MC1 track, meaning models were, in aggregate, no better than chance at assigning their single highest confidence to the actually true answer among a set of plausible-sounding options.
Section 05What actually works, measured
Documented, benchmarked mitigations
Retrieval-Augmented Generation. Shuster et al. (2021), in “Retrieval Augmentation Reduces Hallucination in Conversation,” evaluated knowledge-grounded dialogue on the Wizard of Wikipedia benchmark. An unaugmented BART-Large baseline showed a 68.2% hallucination rate, with only 34.1% of responses actually knowledgeable. Adding Fusion-in-Decoder retrieval, pulling in 5 retrieved documents per response, dropped the hallucination rate to 7.9%, an 8.6x reduction. Grounding a model in retrieved documents converts the generation task from ungrounded parametric recall into something closer to constrained summarization, directly targeting extrinsic hallucination.
Self-consistency sampling. Wang et al. (2022), in “Self-Consistency Improves Chain of Thought Reasoning in Language Models,” replaced greedy decoding with temperature sampling to generate multiple independent reasoning paths, then took a majority vote over the final answers. On PaLM-540B, standard greedy chain-of-thought reached 56.5% accuracy on GSM8K; self-consistency sampling raised that to 74.4%, a 17.9-point gain. On SVAMP, accuracy rose from 79.0% to 86.6%, and on AQuA, from 35.8% to 48.3%. This technique works because it marginalizes over the model’s own stochastic decoding errors: a hallucinated intermediate step that appears in only some sampled paths gets outvoted by the paths that reasoned correctly.
Citation-forcing and attribution verification. Gao et al. (2023) introduced ALCE, a benchmark requiring models to append explicit source citations to every generated claim, then verifying those citations with Natural Language Inference models checking whether the cited source actually supports the claim. Forcing this citation structure constrains the model’s cross-attention to stay anchored on retrieved context tokens during decoding, directly suppressing extrinsic hallucination, and measurably increased user trust scores in downstream evaluations.
Internal activation steering. Mechanistic interpretability techniques, including Inference-Time Intervention (Li et al., 2023) and Adaptive Activation Steering (2024), identify internal directions in a model’s hidden activations that correlate with truthful versus false outputs, using linear probes, and shift the model’s residual stream toward the truthful direction during generation, without any weight updates. On LLaMA-7B, applying Adaptive Activation Steering raised TruthfulQA MC1 accuracy from 23.1% to 42.3%, and raised True * Info from 24.0% to 58.0%.
Folk techniques with no measured effect
| Technique | Mechanism | Measured impact | Status |
|---|---|---|---|
| Retrieval-Augmented Generation | Injects retrieved documents into context | 8.6x reduction in hallucination rate (68.2% → 7.9%) | VERIFIED |
| Self-consistency sampling | Samples multiple reasoning paths, majority vote | +17.9 points on GSM8K (56.5% → 74.4%) | VERIFIED |
| Activation steering | Shifts hidden activations toward truth-correlated directions | +19.2 points on TruthfulQA MC1 (23.1% → 42.3%) | VERIFIED |
| Citation-forcing (ALCE) | Forces per-claim citations verified by NLI | >85% citation precision; measurable trust gains | VERIFIED |
| System-prompt scolding (“do not lie”) | Appends a negative instruction to the system prompt | No statistically significant reduction in extrinsic hallucination | UNVERIFIED, folk advice |
| Setting temperature to 0.0 | Forces greedy, deterministic decoding | Removes sampling variance, but does not correct the underlying parametric errors or exposure bias | UNVERIFIED, folk advice |
| Financial or threat framing (“I will tip $200”) | Appends social pressure to the prompt | No measurable improvement on TruthfulQA or similar benchmarks | UNVERIFIED, folk advice |
The pattern across the verified column is consistent: every technique that actually moves the needle does so by changing what information the model has access to (retrieval, citation-forcing) or by adding an external check on the model’s output (self-consistency voting, activation steering), rather than by asking the model, through instructions alone, to simply be more truthful. Telling a model not to lie does not change the underlying loss landscape from Section 2 that made the false completion cheaper in the first place.
Conclusion
Every result in this lesson traces back to the same root cause worked through directly in Section 2: cross-entropy loss rewards statistical imitation, and a popular falsehood in the training corpus is, mathematically, cheaper for the model to predict than a less common truth. A model’s own reported confidence cannot correct for this, because miscalibration under free generation, compounding exposure bias, and the sycophancy introduced by preference tuning all push logprobs further away from being a reliable truth signal, not closer. TruthfulQA gives this problem an actual number: even capable models fall well short of human accuracy, and smaller models can inflate their score simply by refusing to answer. The mitigations that work, measured here with real benchmark deltas, all share one property: they change what the model can see or add an external check on what it produces, rather than relying on the model to police itself.
The next lesson builds directly on this one. It covers Retrieval-Augmented Generation’s architecture in full depth, the exact mechanism this lesson showed producing an 8.6x reduction in hallucination rate.