From Text Prediction to Chat

Lesson 1 named next-token prediction as the mathematical core of every modern LLM, and covered RLHF as the final step that turns a raw predictor into ChatGPT. This lesson goes back to the middle step: what “predicting the next token” actually optimizes, in formula form, and the two competing research results that changed how every major lab decides how big to make a model and how much text to train it on.

A pretrained model is not trying to be helpful, truthful, or even coherent by design. It is trying to compress its training text as efficiently as possible, and everything that looks like reasoning or knowledge is a side effect of that one narrow objective.

This lesson derives the cross-entropy loss that pretraining actually minimizes, walks through the two rival scaling laws, OpenAI’s 2020 result and DeepMind’s 2022 correction to it, that dictate how a fixed compute budget should be split between model size and training data, and closes with why production models today deliberately move away from what those laws call optimal.

Section 01The pretraining objective, in exact form

A pretrained model performs one task: given a sequence of tokens so far, predict a probability distribution over what token comes next. Let a sequence of T tokens be written X = (x₁, x₂, ..., x_T), where each token xₜ is an integer index into a closed vocabulary V. The joint probability of the entire sequence is decomposed using the chain rule of probability, exactly the autoregressive factorization named in Lesson 1:

x_<t denotes every token before position t, and θ ∈ ℝ^d_θ is the model’s full parameter vector.

To actually compute P(xₜ | x_<t; θ), the transformer stack from Lesson 4 processes x_<t and produces a final d_model-dimensional context vector hₜ ∈ ℝ^d_model. This vector is multiplied by an unembedding matrix W_u ∈ ℝ^(|V| × d_model) to project it into a raw logit score vector zₜ ∈ ℝ^|V|, one score per vocabulary entry:

The logit vector zₜ is converted into an actual probability distribution using softmax, the same operation used inside self-attention in Lesson 4:

Training searches for the parameter set θ that minimizes the empirical cross-entropy loss, L_CE(θ), over a training corpus of T total tokens. This is the same loss family introduced in Lesson 2, applied here across a vocabulary of tens of thousands of classes instead of two:

This is the average negative log-likelihood the model assigns to the actual, ground-truth next token at every position in the training data. Depending on whether the logarithm is natural or base-2, this loss is conventionally reported in nats or in bits per token.

Minimizing this cross-entropy loss is mathematically equivalent to minimizing the Kullback-Leibler (KL) divergence between the true, real-world distribution of language, P_data(xₜ | x_<t), and the model’s learned approximation of it, P_θ(xₜ | x_<t). This is worth stating plainly, because it explains a lot about what a base model is and is not: nothing in this objective directly rewards grammar, logic, or factual correctness. The model never receives a labeled example that says “this sentence is grammatical” or “this fact is true.” Higher-order structure, syntax, reasoning patterns, factual associations, emerges purely because representing that structure turns out to be a more efficient way to compress and predict real text than not representing it. Grammar and fact are learned as a side effect of a narrower, purely statistical compression objective, not as a direct training target.

Section 02Kaplan et al. (2020): the original scaling laws

In 2020, Kaplan et al. at OpenAI published “Scaling Laws for Neural Language Models,” establishing that cross-entropy loss follows predictable power laws as three separate resources scale: the number of non-embedding parameters N, the size of the training dataset in tokens D, and the total training compute in floating-point operations, C. These power laws held across more than seven orders of magnitude of scale. Vocabulary and positional embeddings were deliberately excluded from N, because embedding parameter counts scale with vocabulary size rather than with the transformer’s actual depth or width, and behave differently as loss decreases.

When only one resource is the bottleneck, with the other two assumed unconstrained, loss follows an individual power law in each case:

Parameter-limited regime (enough data, limited parameters): L(N) = (N_c / N)^α_N, with α_N ≈ 0.076 and N_c ≈ 8.8 × 10¹³ parameters.

Data-limited regime (enough parameters, limited data): L(D) = (D_c / D)^α_D, with α_D ≈ 0.095 and D_c ≈ 5.4 × 10¹³ tokens.

Compute-limited regime (both optimally allocated, limited compute): L(C_min) = (C_c^min / C_min)^α_Cmin, with α_Cmin ≈ 0.050 and C_c^min ≈ 3.1 × 10⁸ petaflop-days (1 petaflop-day ≈ 8.64 × 10¹⁹ floating-point operations).

How Kaplan et al. recommended splitting a compute budget

To decide how to split a fixed compute budget C between parameters N and data D, Kaplan et al. used the standard approximation for training compute:

Solving the joint minimization problem for how L(N, D) behaves under this constraint, they found that as compute scales up, parameter count should grow substantially faster than dataset size:

Equating these two relationships gives the structural ratio between how fast N and D need to grow relative to each other:

This means that for every 10x increase in dataset size, compute-optimal training, by Kaplan’s numbers, required scaling parameter count by roughly 10^2.7 ≈ 501x. In practical terms, Kaplan et al.’s recommendation was to put roughly 73% of any additional compute growth into more parameters, and only 27% into more training tokens. This finding shaped an entire generation of frontier model development: labs built enormous parameter counts and stopped training well before the models had seen anywhere near as much text as their parameter count could, in principle, absorb.

Section 03The Chinchilla correction: Hoffmann et al. (2022)

In 2022, Hoffmann et al. at DeepMind published “Training Compute-Optimal Large Language Models,” introducing the Chinchilla scaling framework as a direct empirical correction to Kaplan et al.’s conclusion. They identified a specific methodological problem in the earlier study: Kaplan et al. held learning rate schedules constant, or used suboptimal decay settings, across runs of different sequence lengths, which artificially penalized smaller models trained on longer token sequences and skewed the resulting power-law fit.

Hoffmann et al. trained over 400 transformer models, ranging from 70 million to 16 billion parameters, across token budgets from 5 billion to 500 billion tokens, independently tuning the learning rate schedule and cosine decay target for every individual run length. They modeled loss using a parametric fit that separates out an irreducible loss floor, E, representing the entropy of natural language itself, which no amount of scaling can push below:

Using isoFLOP profiles across this large model sweep, Hoffmann et al. extracted α ≈ 0.34 and β ≈ 0.28. Under a fixed compute budget, balancing the gradient contribution from N against the contribution from D produces a strikingly different scaling relationship than Kaplan’s:

Both exponents come out equal. This is the headline finding: model size and training data must scale in equal proportion, not the roughly 3-to-1 parameter-heavy split Kaplan et al. recommended. Doubling the compute budget should scale both N and D by the same factor, 2^0.5 ≈ 1.41x each, not scale parameters aggressively while barely touching data.

The 20-tokens-per-parameter ratio

The single operational number that came out of Chinchilla is the compute-optimal token-to-parameter ratio:

To train compute-optimally, a model should consume roughly 20 tokens of training data for every one non-embedding parameter it has.

Applying the Chinchilla ratio to real frontier models trained under Kaplan-era assumptions revealed something stark: nearly every large model built before 2022 was severely undertrained relative to its own parameter count.

Hoffmann et al. proved this claim directly, not just theoretically. They trained Chinchilla, at 70 billion parameters, on 1.4 trillion tokens, using the exact same total training compute budget as Gopher, which had 280 billion parameters trained on only 300 billion tokens. Despite having 4x fewer parameters, Chinchilla significantly outperformed Gopher, GPT-3, and PaLM across a wide range of downstream benchmarks, including MMLU. The extra parameters in the older models were, in a very concrete sense, wasted: that compute would have produced a better model if it had gone into more training data on a smaller network instead.

Section 04Why production models over-train anyway

Chinchilla scaling answers one specific question: given a fixed pretraining compute budget, how do you get the lowest possible loss? But that question has a hidden assumption baked into it, that training compute is the only cost that matters. In real deployment, it is not. A model gets trained exactly once, but it then serves potentially billions of inference requests over its operational lifetime. Once that lifetime serving cost is counted, the picture changes.

The inference cost, in formula form

A single forward pass through a transformer during inference costs approximately:

(This is worth contrasting directly with training cost: a training step needs both a forward pass and a backward pass, and the backward pass costs roughly twice the forward pass, so 2N + 4N = 6N FLOPs per token during training, matching the C ≈ 6ND approximation from Section 2.)

A concrete comparison: two models, one loss target

Consider two hypothetical models an engineering team might choose between, both reaching the same validation loss.

Model A (Chinchilla-optimal): N_A = 70 billion parameters, trained on D_A = 1.4 trillion tokens (a 20:1 ratio). Training cost: C_train ≈ 6 × 70B × 1.4T = 5.88 × 10²³ FLOPs.

Model B (inference-optimal, deliberately over-trained): N_B = 8 billion parameters, trained on D_B = 15 trillion tokens (a 1,875:1 ratio, far beyond Chinchilla-optimal). Training cost: C_train ≈ 6 × 8B × 15T = 7.20 × 10²³ FLOPs, about 1.22x more expensive to train than Model A.

Model B costs modestly more to train. But its inference cost per generated token is where the real difference shows up:

Model B costs only 11.4% as much per generated token as Model A, an 88.6% reduction in serving cost. Model B also needs roughly 8.75x less GPU memory to hold its weights and KV cache, the same KV-cache mechanism covered in Lesson 4, meaning it can serve on smaller GPU clusters with higher throughput. Once a model is being called billions of times, that per-token serving cost reduction overwhelms the modest upfront training cost penalty many times over.

Llama 3 8B: over-training as a deliberate strategy

Meta’s Llama 3 technical report (“The Llama 3 Herd of Models”) makes this trade-off explicit for its 8-billion-parameter model. Chinchilla’s 20:1 ratio calls for roughly 8B × 20 ≈ 160 billion training tokens for a model this size. Meta instead trained it on 15 trillion tokens:

That is roughly 93.75x beyond the Chinchilla-optimal ratio for a model of that size. Meta reported that benchmark performance kept improving log-linearly as training continued, well past the point Chinchilla would call optimal, producing an unusually capable 8-billion-parameter model that competes with older 70-billion-class models, at a fraction of the memory footprint and serving cost those larger models require.

Conclusion

Everything in this lesson traces back to the single cross-entropy objective in Section 1: a model is trained to predict the next token as accurately as possible, nothing more directly. Kaplan et al.’s original scaling laws said the most efficient way to reduce that loss, for a fixed compute budget, was to grow parameters far faster than data. Chinchilla corrected that conclusion with a larger, more carefully controlled experiment, and found the two should scale together, with roughly 20 tokens of data for every parameter. But “most efficient use of training compute” and “cheapest model to actually run in production” are two different optimization targets, and once a model’s real cost is dominated by billions of inference calls rather than one training run, deliberately over-training a smaller model, exactly what Llama 3 8B does at nearly 94x the Chinchilla ratio, becomes the better trade.

RLHF, the step that turns a model trained purely on this next-token objective into something that follows instructions and holds a conversation, was already covered in detail in Lesson 1. The next lesson moves to a different piece of the pipeline entirely: how raw text gets mapped into the vector space a model actually reasons in, starting from word2vec’s original embedding geometry.