Lesson 1 defined machine learning as a search for the parameters that make the fewest mistakes across a set of examples. This lesson opens up that search. It is a deterministic, continuous optimization problem, and every step of it can be computed by hand.
Training is not the model understanding anything. Training is repeatedly measuring how wrong a guess is, then nudging every number inside the model a small step in the direction that makes it less wrong.
This lesson walks through how a dataset gets split before training starts, how “wrong” gets turned into a single number, how that number drives an update to every parameter, and a complete, hand-calculated example of a two-parameter model learning a line, three iterations, every arithmetic step shown.
Section 01Splitting the data: training, validation, and test sets
Before any learning happens, a dataset gets divided into three separate, non-overlapping subsets.
- Training set. The data the optimization algorithm actually touches. Every gradient computed during learning comes from this set, and only this set.
- Validation set. Held out from training. Used during development to check how well the model generalizes, to tune hyperparameters such as the learning rate, and to decide when to stop training.
- Test set. Held out from both training and validation. Evaluated exactly once, after every training decision and every hyperparameter choice is already final. It is the only honest estimate of how the model performs on data it has never influenced in any way.
Why one split is not enough
Fitting parameters only to the training set, with no separate check, invites a specific failure: the optimizer will happily drive training error toward zero by matching noise particular to that exact set of examples, rather than learning the general pattern underneath them. A validation set catches this, because a model that memorized training noise will perform visibly worse on validation data it never touched during optimization.
The same contamination risk applies one level up. If a developer tunes hyperparameters by repeatedly checking performance against the test set, the test set stops being an honest measure. Each round of tuning against it leaks information about that specific set of examples into the model's configuration, so the final reported number overstates how the model will do on genuinely new data. This is why the test set gets touched exactly once, at the very end, and never used to make any decision along the way.
Section 02Turning “wrong” into a number: loss functions
A loss function, written L(θ), measures how far a model's prediction ŷ is from the true label y, where θ stands for the model's full set of tunable parameters. Training searches for the setting of θ that makes L as small as possible, averaged across the training set.
Mean Squared Error, for continuous targets
For regression tasks, where the target is a real number rather than a category, the standard loss is Mean Squared Error (MSE):
N is the number of training examples in the batch, ŷi is the model's prediction for example i, and yi is that example's true label. For a simple linear model, the prediction itself is ŷi = w·xi + b, where w is a weight and b is a bias. Squaring the error before averaging does two things at once: it makes every error positive, so overshooting and undershooting don't cancel out, and it penalizes large errors far more heavily than small ones, since the penalty grows with the square of the mistake, not the mistake itself.
Binary Cross-Entropy, for classification targets
For binary classification, where the target is one of two categories and the prediction ŷi is a probability between 0 and 1, MSE is the wrong tool. The standard loss instead is Binary Cross-Entropy (BCE):
Here yi ∈ {0, 1} is the true class, and ŷi ∈ (0, 1) is the predicted probability that the example belongs to class 1. For each example, exactly one of the two terms inside the brackets is active, since one of yi or (1 − yi) is always zero. If the true label is 1, only the yi · ln(ŷi) term contributes, and it grows sharply negative as ŷi approaches 0, meaning the model is confidently wrong. That is the point of the logarithm: it punishes confident, wrong predictions far more severely than a squared-error term would, which matters for a task where the output is meant to be read as a calibrated probability, not just a number to minimize.
Section 03Optimization: gradient descent and backpropagation
Once loss is a single number, training becomes a search: adjust θ to make that number smaller. Gradient descent does this by repeatedly moving every parameter a small step in the direction that reduces the loss fastest, which is the direction opposite the loss function's gradient.
The theoretical foundation for computing that gradient across a multi-layer network is backpropagation, established by David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams in their 1986 Nature paper, “Learning representations by back-propagating errors.” Backpropagation applies the multivariate chain rule to compute the exact partial derivative of the loss with respect to every single weight and bias in the network, all at once, by propagating the error signal backward from the output layer to the input layer.
The parameter update rule
For a parameter vector θ at iteration t, gradient descent applies:
η (eta) is the learning rate, a hyperparameter that controls how large each step is. ∇θL(θt) is the gradient: a vector holding the partial derivative of the loss with respect to every parameter, [∂L/∂w, ∂L/∂b]T for the two-parameter case used below. Each entry of that vector points in the direction that would increase the loss fastest if the corresponding parameter were increased. Subtracting the gradient, scaled by η, moves every parameter a small step in exactly the opposite direction: the direction that decreases loss fastest, given the current values of every parameter.
This single rule, applied repeatedly, is the entire mechanism behind training a neural network. Everything else, including the vastly more complex networks covered in later lessons, is this same update rule applied to a much larger θ, with backpropagation supplying the gradient through many more layers.
Section 04A fully worked example: a two-parameter linear model
Abstractions aside, here is the complete arithmetic, three iterations, every number shown.
The setup
The model is ŷ = w · x + b, with two learnable parameters: weight w and bias b. The training set has three examples:
| Sample | x | y |
|---|---|---|
| 1 | 1 | 3 |
| 2 | 2 | 5 |
| 3 | 3 | 7 |
The underlying ground-truth relationship generating this data is y = 2x + 1, so the optimal parameters, which gradient descent is trying to discover purely from the three data points, are w* = 2.0 and b* = 1.0. Training never gets to see this equation directly. It only sees the three (x, y) pairs and has to recover the pattern from them.
Using Mean Squared Error over these three points:
Taking the partial derivative of this loss with respect to each parameter gives the two gradient components needed for every update:
∂L/∂b = (2/3) Σi=13 (ŷi − yi)
Set the learning rate η = 0.05. Initialize both parameters to zero: w₀ = 0.0, b₀ = 0.0. Starting at zero is a common, simple initialization: the model begins knowing nothing, predicting the same constant output regardless of input, and gradient descent has to pull it toward the correct line using nothing but the error signal from these three points.
Predictions, using ŷ = w₀·x + b₀ = 0.0·x + 0.0, which is 0.0 for every input regardless of x:
ŷ₂ = 0.0(2) + 0.0 = 0.0
ŷ₃ = 0.0(3) + 0.0 = 0.0
Errors, ei = ŷi − yi:
e₂ = 0.0 − 5 = −5.0
e₃ = 0.0 − 7 = −7.0
Loss:
Gradients:
∂L/∂b = (2/3)[−3.0 + (−5.0) + (−7.0)] = (2/3)(−15) = −10.0000
Parameter update, using θt+1 = θt − η · ∇θL:
b₁ = 0.0 − 0.05 · (−10.0000) = 0.0 + 0.5000 = 0.5000
The updates move both parameters in the correct direction: the true line has w* = 2.0 and b* = 1.0, and after a single step, both parameters moved from zero toward those values, in proportion to how much each one contributed to the total error.
Predictions, using w₁ = 1.1333, b₁ = 0.5000:
ŷ₂ = 1.1333(2) + 0.5000 = 2.7667
ŷ₃ = 1.1333(3) + 0.5000 = 3.9000
Errors:
e₂ = 2.7667 − 5.0 = −2.2333
e₃ = 3.9000 − 7.0 = −3.1000
Loss:
The loss dropped sharply, from 27.6667 to 5.4885, in a single step.
Gradients:
∂L/∂b = (2/3)[−1.3667 − 2.2333 − 3.1000] = (2/3)(−6.7000) ≈ −4.4667
Parameter update:
b₂ = 0.5000 − 0.05 · (−4.4667) = 0.5000 + 0.2233 = 0.7233
Predictions, using w₂ = 1.6378, b₂ = 0.7233:
ŷ₂ = 1.6378(2) + 0.7233 = 3.9989
ŷ₃ = 1.6378(3) + 0.7233 = 5.6367
Errors:
e₂ = 3.9989 − 5.0 = −1.0011
e₃ = 5.6367 − 7.0 = −1.3633
Loss:
Three iterations, side by side
| Iteration | Weight (w) | Bias (b) | MSE Loss (L) | ∂L/∂w | ∂L/∂b |
|---|---|---|---|---|---|
| 0 | 0.0000 | 0.0000 | 27.6667 | −22.6667 | −10.0000 |
| 1 | 1.1333 | 0.5000 | 5.4885 | −10.0889 | −4.4667 |
| 2 | 1.6378 | 0.7233 | 1.0897 | −4.4919 | −1.9963 |
Both parameters move steadily toward the true values, w* = 2.0 and b* = 1.0, and the loss falls by roughly a factor of 5 at each step. Left to continue, this process converges toward the exact ground-truth line, purely from repeating the same update rule against the same three data points.
Section 05Learning rate sensitivity: the same starting point, three outcomes
The learning rate η is not a detail. Using the exact same starting point (w₀ = 0.0, b₀ = 0.0, with initial gradients ∂L/∂w = −22.6667 and ∂L/∂b = −10.0000, both computed above), here is what happens to the very first update under three different values of η.
Learning rate too low: η = 0.001
b₁ = 0.0 − 0.001 · (−10.0) = 0.0100
The resulting loss after this step is L₁ ≈ 26.6083, barely below the starting loss of 27.6667. The parameters moved in the right direction, but by such a small amount that reaching convergence would take tens of thousands of steps, an enormous, avoidable amount of compute spent taking steps that are each too small to matter much.
Optimal learning rate: η = 0.05
This is the value used in the full worked example above: w₁ = 1.1333, b₁ = 0.5000, loss falling steadily and monotonically toward the true minimum, exactly as shown in the three-iteration table.
Learning rate too high: η = 0.50
b₁ = 0.0 − 0.50 · (−10.0) = 5.0000
These parameters have overshot wildly. The true weight is 2.0, and a single step has already pushed w to 11.3333. Recomputing predictions and loss with these new, overshot parameters:
e₁ = 13.3333, e₂ = 22.6667, e₃ = 32.0000
L₁ = (1/3)[(13.3333)² + (22.6667)² + (32.0000)²] ≈ (177.7778 + 513.7778 + 1024.0000)/3 ≈ 571.8519
One step turned a loss of 27.6667 into a loss of 571.8519, a 20-fold increase in a single update. This is numerical divergence: each subsequent step overshoots further than the last, and the parameters typically blow up toward infinity or produce invalid (NaN) values within a handful of iterations. Too small a learning rate wastes compute. Too large a learning rate destroys the optimization entirely.
Section 06Modern optimizers: why almost nobody uses plain gradient descent
Basic gradient descent, as used in the worked example, applies one single, fixed learning rate to every parameter, at every step, for the entire run. Modern deep learning training almost universally uses an adaptive alternative instead: Adam (Adaptive Moment Estimation), published by Diederik P. Kingma and Jimmy Ba in 2014.
Adam tracks two running exponential moving averages for every parameter, updated at each step t using the current gradient gt:
vt = β2vt−1 + (1 − β2)gt²
mt is the first moment, a smoothed running average of the gradient itself, functioning like momentum: it keeps moving in a direction that has been consistently useful over recent steps, rather than reacting fully to a single noisy gradient. vt is the second moment, a smoothed running average of the squared gradient, which tracks how large and how volatile the gradient has recently been for that specific parameter. β1 and β2 are decay-rate hyperparameters, both in (0, 1), typically set to 0.9 and 0.999 respectively, controlling how much weight recent gradients carry relative to older ones.
Because both moving averages start at zero, they are biased toward zero during the earliest steps of training. Adam corrects for this with a bias-corrected version of each:
The final parameter update then becomes:
ε is a tiny constant added purely to prevent division by zero. The practical effect of this formula is that each parameter gets its own effective learning rate, scaled down automatically for parameters whose recent gradients have been large or noisy (large v̂t shrinks the step), and scaled up, relatively, for parameters with small, stable gradients. This is why Adam tends to converge reliably across a much wider range of initial learning rate choices than plain gradient descent, which has no such per-parameter adjustment and depends entirely on a single, globally-tuned η.
Section 07Reading loss curves: catching overfitting before it happens
Monitoring both training loss and validation loss across training gives a direct, visual signal for when a model has learned enough and when it has started learning too much of the wrong thing.
- Underfitting phase. Training loss and validation loss both drop together. The model is still discovering genuine, generalizable patterns that show up in both partitions of the data.
- Optimal convergence point. Validation loss reaches its lowest point. This is the ideal moment to stop training, a technique called early stopping.
- Overfitting phase. Training loss keeps falling, often all the way toward zero, while validation loss stops falling and starts rising.
Why the divergence is the signal, not just a curiosity
Once training loss and validation loss diverge, the model's remaining capacity is being spent memorizing noise, outliers, and quirks specific to the training examples rather than the underlying pattern connecting inputs to outputs. Because that memorized noise does not exist in the validation set, predictions on validation examples get worse even as predictions on training examples keep improving. The gap between the two curves is a direct, measurable readout of how much the model has shifted from learning a general pattern to memorizing specific training points. Watching for that gap, and stopping before it grows, is one of the simplest and most effective tools for keeping a trained model useful on data it has never seen.
Conclusion
Every step in this lesson chains into the next one. A dataset gets split so that a model's real performance can be measured honestly. A loss function turns “how wrong is this prediction” into a single number. Gradient descent uses that number's gradient to nudge every parameter toward less error, one small step at a time, and the size of that step is not a minor detail: too small wastes enormous compute, too large can destroy the optimization outright, as the 20-fold loss spike in this lesson's own example showed directly. Adam refines this same basic update rule so that each parameter gets its own adaptive step size, and watching training loss against validation loss tells you exactly when to stop. None of this is guesswork. It is arithmetic, repeated automatically, at a scale no person could do by hand, on a problem far larger than three data points. But the mechanism is exactly the one worked through here.
The next lesson builds on this one directly. It opens up what happens inside each of the stacked layers that make a network “deep,” and why removing the nonlinearity between those layers would collapse the entire structure back down into something no more powerful than the single-layer model used in this lesson's worked example.