Neural Networks, Conceptually

Lesson 2 trained a single linear model, one weight, one bias, using gradient descent. A neural network is built from many of those same linear pieces, stacked in layers. But stacking linear pieces alone changes nothing: a network with a hundred layers of pure linear math is mathematically identical to a network with one layer, unless something breaks the pattern between the layers.

A neural network is not powerful because it is deep. It is powerful because something non-linear sits between every pair of stacked linear layers, and that one addition is what lets depth mean anything at all.

This lesson builds a network from a single neuron up to a full feedforward layer, proves exactly why stacking linear layers alone collapses to nothing more than one linear layer, walks through the theorem that justifies adding non-linearity, compares the three activation functions used in production models today, and finishes with a complete, hand-calculated forward and backward pass through a real two-layer network.

Section 01The single neuron: three operations, one output

The basic unit of a neural network is a single artificial neuron. It takes an n-dimensional input vector x = [x₁, x₂, ..., xₙ]ᵀ and produces one scalar output, through three sequential steps.

1. Weighted scaling. Each input feature xᵢ is multiplied by its own adjustable weight wᵢ, giving wᵢxᵢ.
2. Bias offset. A single adjustable bias b is added to the sum of all the weighted inputs. The bias lets the neuron’s activation threshold shift independently of whatever the input values happen to be, the same role the bias played for the single linear model in Lesson 2.
3. Non-linear activation. The weighted, biased sum, called the pre-activation value z, is passed through a non-linear function σ to produce the neuron’s final output a.

Written algebraically, with w = [w₁, w₂, ..., wₙ]ᵀ as the neuron’s weight vector:

This is exactly the linear model from Lesson 2, wᵀx + b, with one addition: the result gets passed through σ before it counts as the neuron’s output. Everything in the rest of this lesson is this one operation, repeated and stacked.

Section 02From one neuron to a layer: vectorizing with matrices

Real networks do not use one neuron at a time. A fully connected layer, also called a dense layer, groups m neurons together, and every one of those m neurons receives the exact same n-dimensional input vector x.

Computing m separate dot products one at a time works, but it throws away the parallelism that makes modern hardware fast. Instead, the m neurons’ weight vectors are stacked as the rows of a single weight matrix W ∈ ℝ^m×n, where row j of W is the transposed weight vector of neuron j. The m individual biases become a single bias vector b ∈ ℝᵐ. The entire layer’s computation becomes one matrix-vector multiplication:

z is the pre-activation vector, holding all m neurons’ weighted sums at once. σ is applied element-wise, meaning each of the m entries in z gets passed through the same activation function independently, producing the output vector a.

Batching: many samples at once

Training rarely processes one input at a time either. When a minibatch of B independent samples is processed together, the B input vectors get stacked into a single data matrix X ∈ ℝ^B×n, and the layer computation becomes:

Here 1_B ∈ ℝ^B×1 is a column of ones, whose only job is to broadcast the single bias vector b across every one of the B rows, so each sample in the batch gets the same bias added. Z ∈ ℝ^B×m holds every sample’s pre-activation values, and A ∈ ℝ^B×m holds every sample’s final activations. This is why GPUs and TPUs are so effective for this kind of workload: the entire batch’s worth of neuron computations reduces to a small number of large matrix multiplications, which is exactly the operation that kind of hardware is built to run in parallel.

Section 03The collapse proof: why non-linearity is not optional

Here is the claim this lesson opened with, proven directly. Take a network built from L sequential layers, but strip out every non-linear activation function, so each layer is a pure linear (technically affine) transformation:

...continuing the same pattern up to the final layer:

Substitute the first layer’s output directly into the second layer’s equation:

Define an effective weight matrix W_eff = W⁽²⁾W⁽¹⁾ and an effective bias vector b_eff = W⁽²⁾b⁽¹⁾ + b⁽²⁾. The two-layer network reduces to:

One linear layer. Two stacked linear layers produced nothing that a single linear layer could not produce on its own, because matrix multiplication is closed under linear operations: multiplying matrices together always produces another matrix, never something structurally new. Extending this same substitution recursively through all L layers proves the general case:

No matter how many layers L is, the entire stack of linear transformations always collapses algebraically into a single affine transformation, y = W_collapsed x + b_collapsed. A 100-layer network with no non-linearity anywhere in it cannot represent any decision boundary a single linear layer could not already represent. Depth, by itself, buys nothing. What buys something is breaking this collapse, and that is exactly what a non-linear activation function does: because σ(Wx + b) is not, in general, expressible as W’x + b’ for any single matrix W’ and vector b’, inserting σ between layers stops the substitution trick above from working at all. Each layer becomes a genuinely new transformation, not a redundant repetition of the one before it.

Section 04The Universal Approximation Theorem: what non-linearity buys you

The formal justification for using non-linear activations is the Universal Approximation Theorem, first proven by George Cybenko in 1989 for continuous sigmoidal activation functions, and extended by Kurt Hornik in 1991 to essentially any non-polynomial activation function.

Informally, the theorem states that a feedforward network with a single hidden layer, containing a finite number of neurons, using almost any continuous non-linear activation function, can approximate any continuous function f: ℝⁿ → ℝᵐ on a compact (closed and bounded) region of its input space to arbitrary precision ε > 0. Given enough neurons in that one hidden layer, the network’s output can be made to differ from the true function by as little as you like, anywhere in that region.

This is the theoretical payoff for the collapse proof in Section 3. Non-linear activation functions are precisely what gives a network the expressive capacity to bend a decision boundary into a genuinely complex, curved surface, rather than staying stuck as a single flat hyperplane. The theorem does not say a single hidden layer is the practical way to build a good model. It says the representational capacity is there in principle, which is why the field bothered developing deep, many-layer architectures in the first place: depth turns out to be a far more parameter-efficient way to reach that same expressive power than making one hidden layer arbitrarily wide.

Section 05Three activation functions used in production models today

Not every non-linear function makes a good activation function in practice. The choice affects how well gradients flow backward through a deep network during training, which is the mechanism from Lesson 2 that actually drives learning.

Rectified Linear Unit (ReLU)

Introduced for deep networks by Nair and Hinton in 2010, ReLU is close to the simplest possible non-linear function:

Its derivative is piecewise constant: f’(x) = 1 for x > 0, and f’(x) = 0 for x < 0. At x = 0 the derivative is technically undefined, and frameworks handle this by simply assigning it 0 or 1 by convention.

Because the gradient for any positive input is exactly 1, with no shrinking or saturating, ReLU avoids the vanishing gradient problem that plagued older, smooth activation functions like sigmoid and tanh for positive pre-activations. It also produces sparse activations, since any negative input produces an output of exactly zero. The tradeoff is the “dying ReLU” failure mode: a neuron whose pre-activation lands in negative territory during training produces a gradient of exactly zero, which means it stops receiving any update at all, and can remain permanently inactive for the rest of training.

Gaussian Error Linear Unit (GELU)

GELU weighs an input by its percentile under the standard Gaussian cumulative distribution function Φ(x) = P(X ≤ x), where X ~ N(0,1):

In production libraries such as Hugging Face Transformers, this is commonly approximated with a closed-form expression using tanh, since the exact erf-based formula is more expensive to compute:

Unlike ReLU, GELU is smooth and non-monotonic, with a slight negative curvature just below zero, meaning it produces small negative outputs for small negative inputs instead of hard-clipping them to exactly zero. This lets small gradients continue to flow backward even from mildly negative pre-activations, which helps avoid the dying-neuron failure mode ReLU can hit. GELU is the standard activation function inside modern Transformer architectures, including GPT-2, GPT-3, and BERT.

Sigmoid Linear Unit (SiLU / Swish)

SiLU multiplies its input by the standard logistic sigmoid of that same input:

Its derivative works out to:

Like GELU, SiLU is smooth and non-monotonic, and it is bounded below by a small negative value near x ≈ −0.278 rather than clamping to exactly zero. This preserves small gradients for negative inputs while still preventing the vanishing-gradient problem for positive ones. SiLU is the standard activation function in several modern large language model families, including Llama and Mistral.

Section 06Measuring depth in real architectures

“Depth” means the count of sequential, parameterized transformation layers stacked between the raw input and the final output. Depth lets a network build a hierarchy of representations: earlier layers tend to extract simple, low-level patterns, and later layers combine those into increasingly abstract features, though the boundary between “low-level” and “abstract” gets far less clean in practice than that description suggests.

Two real architectures illustrate the range in use today. OpenAI’s GPT-2 (Radford et al., 2019, “Language Models are Unsupervised Multitask Learners”) was released in four sizes, with depth (n_layer) and hidden embedding width (n_embd) both scaling up together:

In computer vision, ResNet-152 (He et al., 2015, “Deep Residual Learning for Image Recognition”) reaches 152 sequential convolutional layers, using 60.2 million parameters. A network this deep runs into a real, separate problem from the collapse proof above: even with non-linearity present at every layer, gradients computed via backpropagation, covered next, tend to shrink as they get multiplied backward through dozens or hundreds of layers. ResNet’s answer to this is the residual skip connection, which lets the gradient signal bypass a block of layers entirely rather than being forced through every one of them, preventing the gradient from degrading to near zero by the time it reaches the earliest layers.

Section 07Forward pass and backward pass: the two halves of training a network

Running a neural network involves two distinct passes.

1. The forward pass. Input data flows through the network, layer by layer, producing a final prediction and, when a true label is available, a scalar loss value, using the loss functions defined in Lesson 2.
2. The backward pass, or backpropagation. The gradient of that scalar loss flows in reverse, from the output back toward the input, using the multivariable chain rule to compute the exact partial derivative of the loss with respect to every single weight and bias in the network.

The general backpropagation formulation

For an L-layer network, each layer l computes:

with a⁽⁰⁾ = x as the raw input and a^(L) = ŷ as the network’s final prediction. A scalar loss function L(ŷ, y) measures error against the target y.

To get the gradients needed for the parameter update rule from Lesson 2, backpropagation defines an error delta vector at each layer, δ^(l) = ∂L/∂z^(l), and computes it using three expressions.

Output layer error, using ⊙ for the Hadamard (element-wise) product:

Recursive hidden layer error, propagating the error backward one layer at a time:

Parameter gradients, computed directly from each layer’s delta:

Every gradient the parameter update rule in Lesson 2 needs comes directly out of this recursive backward sweep, computed once per layer, reusing the delta from the layer just processed instead of recomputing anything from scratch.

Section 08A fully worked example: forward and backward pass through a real network

Here is the entire mechanism, every number shown, for a concrete two-layer feedforward network.

Architecture

• Input: x = [1.0, 2.0]ᵀ ∈ ℝ²
• Layer 1 (hidden): 2 neurons, logistic sigmoid activation σ(z) = 1/(1+e⁻ᶻ). Weights W⁽¹⁾ = [[0.5, −0.2], [0.8, 0.3]], biases b⁽¹⁾ = [0.1, −0.4]ᵀ
• Layer 2 (output): 1 neuron, logistic sigmoid activation. Weights W⁽²⁾ = [0.4, −0.6], bias b⁽²⁾ = 0.2
• Target output: y = 1.0
• Loss function: Mean Squared Error, L = (1/2)(ŷ − y)²

Step 1: forward pass

z⁽¹⁾ = W⁽¹⁾x + b⁽¹⁾

z₁⁽¹⁾ = (0.5 × 1.0) + (−0.2 × 2.0) + 0.1 = 0.5 − 0.4 + 0.1 = 0.2

z₂⁽¹⁾ = (0.8 × 1.0) + (0.3 × 2.0) + (−0.4) = 0.8 + 0.6 − 0.4 = 1.0

a₁⁽¹⁾ = σ(0.2) = 1/(1+e⁻⁰·²) ≈ 0.549834

a₂⁽¹⁾ = σ(1.0) = 1/(1+e⁻¹·⁰) ≈ 0.731059

z⁽²⁾ = W⁽²⁾a⁽¹⁾ + b⁽²⁾ = (0.4 × 0.549834) + (−0.6 × 0.731059) + 0.2 ≈ −0.018702

ŷ = a⁽²⁾ = σ(−0.018702) = 1/(1+e⁰·⁰¹⁸⁷⁰²) ≈ 0.495325

L = (1/2)(0.495325 − 1.0)² = (1/2)(−0.504675)² ≈ 0.127349

The network’s prediction of roughly 0.495 falls well short of the target 1.0, giving a substantial starting loss. The backward pass now computes exactly how much each of the six parameters (four weights and two biases in layer 1, two weights and one bias in layer 2) contributed to that error.

Step 2: backward pass

Output loss derivative, ∂L/∂a⁽²⁾ = a⁽²⁾ − y (the derivative of the MSE loss with respect to the prediction):

∂L/∂a⁽²⁾ = 0.495325 − 1.0 = −0.504675

Output local activation derivative, using σ’(z) = σ(z)(1 − σ(z)):

∂a⁽²⁾/∂z⁽²⁾ = 0.495325 × (1 − 0.495325) ≈ 0.249978

Layer 2 error delta, δ⁽²⁾ = (∂L/∂a⁽²⁾) × (∂a⁽²⁾/∂z⁽²⁾):

δ⁽²⁾ = −0.504675 × 0.249978 ≈ −0.126158

Layer 2 parameter gradients, using ∂L/∂W⁽²⁾ = δ⁽²⁾(a⁽¹⁾)ᵀ:

∂L/∂W⁽²⁾ = −0.126158 × [0.549834, 0.731059] = [−0.069366, −0.092229]

∂L/∂b⁽²⁾ = δ⁽²⁾ = −0.126158

Backpropagate the error into layer 1, using (W⁽²⁾)ᵀδ⁽²⁾:

∂L/∂a⁽¹⁾ = [0.4, −0.6]ᵀ × (−0.126158) = [−0.050463, 0.075695]ᵀ

σ’(z₁⁽¹⁾) = 0.549834 × (1 − 0.549834) ≈ 0.247517

σ’(z₂⁽¹⁾) = 0.731059 × (1 − 0.731059) ≈ 0.196612

Layer 1 error delta, δ⁽¹⁾ = (∂L/∂a⁽¹⁾) ⊙ σ’(z⁽¹⁾):

δ⁽¹⁾ = [−0.050463 × 0.247517, 0.075695 × 0.196612]ᵀ ≈ [−0.012490, 0.014882]ᵀ

Layer 1 parameter gradients, using ∂L/∂W⁽¹⁾ = δ⁽¹⁾xᵀ:

∂L/∂W⁽¹⁾ = [−0.012490, 0.014882]ᵀ × [1.0, 2.0] = [[−0.012490, −0.024981], [0.014882, 0.029765]]

∂L/∂b⁽¹⁾ = δ⁽¹⁾ = [−0.012490, 0.014882]ᵀ

The complete result

Every one of these six gradient values plugs directly into the update rule from Lesson 2, θ_t+1 = θ_t − η·∇_θL, to produce the next iteration’s parameters. A real network repeats exactly this process, millions or billions of times, across millions or billions of parameters instead of six.

Section 09Parameters vs. hyperparameters

Two categories of numbers govern a trained network, and confusing them is a common source of confusion in production configuration.

Parameters are the internal, learnable values stored inside the network’s weight and bias tensors. They start out randomly initialized, commonly from a truncated normal distribution (GPT-2 uses a standard deviation of 0.02), and every one of them gets updated automatically during training through the backpropagated gradients computed above. Weight matrices, bias vectors, token embedding tables, positional embedding matrices, and layer-normalization gains and biases are all parameters.

Hyperparameters are structural and algorithmic settings an engineer fixes before training starts. They define the shape of the computational graph itself, constrain how many parameters the model even has, and control how optimization behaves, but they are never touched by a backpropagation gradient. Layer count, embedding dimension, attention head count, maximum context length, vocabulary size, the learning rate, and dropout probability are all hyperparameters.

GPT-2 Small’s own published configuration makes the split concrete. Its hyperparameters are n_layer = 12, n_embd = 768, n_head = 12, vocab_size = 50257, n_positions = 1024, initializer_range = 0.02, and activation_function = “gelu_new”. Those fixed hyperparameter choices determine exactly how many learnable parameters get instantiated in memory: the token embedding matrix alone holds 50,257 × 768 = 38,597,376 parameters, the position embedding matrix holds 1,024 × 768 = 786,432 parameters, a single attention projection weight matrix holds 768 × (3 × 768) = 1,769,472 parameters, and a single MLP expansion layer weight holds 768 × 3,072 = 2,359,296 parameters. Every one of those numbers is a direct, arithmetic consequence of the hyperparameters chosen before a single training step ever ran.

Conclusion

A neural network is matrix multiplication, an added bias, and a non-linear function, repeated in layers. The collapse proof in this lesson is the reason that description is not an oversimplification: strip out the non-linearity and every one of those stacked layers algebraically reduces to exactly one linear layer, no matter how many you stack. The Universal Approximation Theorem is the payoff for keeping the non-linearity in: it is what gives a network, even a shallow one, the theoretical capacity to represent essentially any continuous function. ReLU, GELU, and SiLU are three different, production-proven ways to implement that non-linearity, each with its own tradeoff between gradient flow and computational simplicity. And the full forward-and-backward pass worked through by hand in Section 8 is not a simplified stand-in for what happens inside a real model. It is the exact mechanism, at a scale of six parameters instead of billions.

The next lesson builds directly on this one. It introduces the transformer’s self-attention mechanism, which replaces the fixed, position-by-position weight matrices used in this lesson’s dense layers with weights computed dynamically from the input itself, a change that turned out to remove the sequential bottleneck that had limited every earlier architecture for processing language.