How a Neural Network Learns

Introduction

A neural network is a function. You give it numbers (the pixels of an image, the words of a sentence turned into vectors, the measurements of a flower) and it gives back numbers: a prediction $\hat y$. The hat means predicted: the network’s answer, which we compare with the true answer $y$.

What makes it a learning machine is that this function has knobs: thousands, sometimes billions of numbers, the weights and the biases, that decide what the function computes. We write them all in one big vector $\theta$, and the network as $\hat y = f_\theta(x)$.

Training the network means finding values of the knobs for which its predictions are close to the true answers, on examples where we know them. We measure “close” with a single number, the loss: the bigger it is, the worse the network. Training is then a minimization problem:

For a model with two knobs, like a straight line $\hat y = a x + b$, there is a formula that gives the best $(a, b)$ directly. For a network with millions of knobs there is none. Instead, we improve the knobs a little at a time:

Each arrow answers the same question: in which direction should I turn each knob, and by how much, to lower the loss? This page builds the answer in four steps:

1. The derivative tells how much a function’s output changes when you turn one knob.
2. The chain rule computes that derivative for a function made of layers, which is what a network is: this is backpropagation.
3. Gradient descent uses the derivative to walk downhill, one step at a time, and the learning rate sets the size of the steps.
4. The gradient does the same for many knobs at once.

At the end, you will train a small model by hand, with pen and paper, and watch a real network train in your browser. All you need is to remember what a function is; section 0 recalls, quickly, the two other objects we need, vectors and matrices, and uses them to write what a network computes: the forward pass.

0. Vectors, matrices and the forward pass

0.1 Data are vectors

A vector is a list of numbers. In machine learning, every data point is one: an iris flower is the vector of its four measurements (sepal length and width, petal length and width), $x \in \mathbb{R}^4$; a word, for a language model, is a vector of a few hundred numbers, its embedding.

Vectors add entry by entry, and multiply by a number entry by entry. The operation that matters most is the dot product, which multiplies two vectors entry by entry and sums the results:

The length, or norm, of a vector is $\|u\| = \sqrt{u \cdot u}$, and dividing the dot product by the two lengths gives the cosine similarity, the cosine of the angle between the two vectors:

It is $1$ when the vectors point the same way, $0$ when they are unrelated, $-1$ when they are opposite. This is how search engines and language models compare documents or words: two vectors that point the same way mean similar things.

0.2 A layer is a matrix

A matrix of size $m \times n$ is a table of numbers with $m$ rows and $n$ columns. Multiplied by a vector of $\mathbb{R}^n$, it gives a vector of $\mathbb{R}^m$: each entry of the result is the dot product of one row of the matrix with the vector.

The sizes must fit: an $(m \times n)$ matrix times an $(n \times 1)$ vector gives an $(m \times 1)$ vector, and the two inner sizes, $n$, must be equal.

This product is exactly what a layer of a network computes. A layer with $n$ inputs and $m$ neurons multiplies its input $x$ by a weight matrix $W$ of size $m \times n$, adds a bias vector $b$, then applies an activation $\sigma$ to each entry:

$W$ and $b$ are the knobs of the layer: with 2 inputs and 3 neurons, $W$ is $3 \times 2$ and $b$ has 3 entries, so 9 knobs. A network stacks several such layers, the output of one being the input of the next.

0.3 The forward pass

Running a network on an input means computing its layers one after the other, from the input to the output: this is the forward pass. With one hidden layer and one output, it is two lines:

The hidden layer turns the input $x$ into a vector $h$, one entry per neuron; the output layer takes a weighted sum of these entries, with no activation. The knobs of the network are $\theta = (W_1, b_1, W_2, b_2)$.

For example, with 2 inputs, 2 hidden neurons and the activation $\text{ReLU}(z) = \max(0, z)$, which keeps the positive numbers and replaces the negative ones by $0$:

The second neuron receives a negative number and outputs $0$: for this input, it plays no part in the prediction.

With a single input $x$, the sizes shrink: $W_1$ is a column of $n$ weights $w_j$, $W_2$ a row of $n$ weights $v_j$, and $b_2$ a single number $c$. The network becomes a function from $\mathbb{R}$ to $\mathbb{R}$,

and a function from $\mathbb{R}$ to $\mathbb{R}$ can be drawn. Do the forward pass for every $x$ and you get a curve: the shape the network draws. With ReLU, each neuron adds one bend to it, and its knobs decide where the bend is and how steep. More neurons, more bends: with enough of them, a network can draw almost any shape.

1. The derivative: what happens when you turn one knob

1.1 A rate of change

Drive 120 km in 2 hours and your average speed is 60 km/h: the change of distance divided by the change of time. Your speedometer shows something else, the speed now: the same ratio, measured over a moment so short that the speed has no time to change.

The derivative is that speedometer, for any function. Between $x_0$ and $x_0 + h$, the function $f$ changes by $f(x_0 + h) - f(x_0)$, so its average rate of change is

Make $h$ smaller and smaller. If this ratio settles on a finite value, $f$ is differentiable at $x_0$, and that value is its derivative:

It is also written $\dfrac{df}{dx}$: a tiny change of $f$, $df$, divided by the tiny change of $x$, $dx$, that caused it. On the graph of $f$, it is the slope of the tangent at $x_0$.

For $f(x) = x^2$, with $a^2 - b^2 = (a + b)(a - b)$:

So $f'(x_0) = 2x_0$, at every point: the derivative is itself a function, $f'(x) = 2x$.

1.2 In practice: a table and a few rules

Nobody computes limits by hand every time. We use a table for the usual functions, and rules to combine them.

The sign of $f'(x_0)$ says which way the function goes at $x_0$: up if it is positive, down if it is negative. Its size says how fast. Keep this in mind: it is all a computer needs to find which way is down.

2. The chain rule: derivatives through the layers

2.1 A network is a chain of functions

Each layer of a network is a function: it takes the output of the previous layer and computes its own. The network is the composition of its layers,

with $f_1$ applied first. To train it, we need the derivative of such a composed function, and the table of section 1 has no line for it. The chain rule fills the gap. For two functions:

In words: the derivative of the inside, times the derivative of the outside evaluated at the inside.

For example, $k(x) = e^{(3x + 1)^2}$ is three layers: $h(x) = 3x + 1$, then $g(x) = x^2$, then $f(x) = e^x$. One factor per layer, each evaluated at the input of its layer:

2.2 With n layers: forward, then backward

Name the intermediate values of a chain of $n$ layers: $u_0 = x$, $u_1 = f_1(u_0)$, …, $u_n = f_n(u_{n-1}) = \hat y$. The chain rule gives

The order of the computation follows: first go forward and compute every $u_k$ (the forward pass of section 0.3), since each factor needs the input of its layer; then go backward, from the output to the input, multiplying the factors as you go. Every layer reuses the product computed by the layer after it: nothing is computed twice.

2.3 Backpropagation

Now the knobs. The loss compares the prediction with the true answer, for example $\mathcal{L} = (\hat y - y)^2$, and depends on every weight. To know which way to turn one weight $w$, keep all the others fixed and look at how the loss changes:

The round $\partial$ instead of $d$ says that we differentiate with respect to one variable, the others being fixed: a partial derivative, more in section 4. Its sign says in which direction the loss changes when $w$ goes up; its size says by how much.

Try it on a network with a single knob: $\hat y = w\,x$ with $x = 2$, $y = 1$, and $w = 1$. Turning the knob by hand from $w = 1$ to $w = 1.01$ moves $\hat y$ from $2$ to $2.02$ and the loss from $1$ to $1.0404$: a rate of change of $\frac{0.0404}{0.01} \approx 4$. The chain rule gives the same number, exactly and without trying:

The derivative is positive: turning $w$ up makes the loss worse, so we should turn it down, in the direction opposite to the derivative.

Backpropagation is this computation done for every weight of a network at once: the chain rule applied recursively, from the loss back to each parameter. One forward pass computes the prediction and the loss; one backward pass gives the derivative of the loss with respect to every weight.

3. Walking downhill: gradient descent

3.1 Where is the minimum?

At the bottom of a valley, the ground is flat: at a minimum of a smooth function, the tangent is horizontal and $f'(x) = 0$. So the solutions of $f'(x) = 0$ are the candidates. Only candidates, though: $x^3$, $1 - x^2$ and $x^2 - 1$ all have $f'(0) = 0$, and the first one has no extremum at $0$, it only pauses before going up again.

A minimum can also be local (the lowest point around it) or global (the lowest point everywhere). A curve with two valleys has two local minima, and only the deeper one is global.

All of this needs the expression of $f$, and equations we can solve. The loss of a network has millions of variables, and no one can solve $\mathcal{L}'(\theta) = 0$ for it. We need a method that only looks at the function where we are.

3.2 One step at a time

Stand on a hillside in the fog. You can’t see the valley, but you can feel the slope under your feet, so you take a step downhill, feel the slope again, take another step. The derivative is that slope: if $f'(x) > 0$, the function goes up to the right, so the way down is to the left; if $f'(x) < 0$, it is to the right. In both cases, move against the sign of the derivative:

This is gradient descent. The number $\eta > 0$ is the learning rate, or step size: it sets how far each step goes. And since $f'(x_n)$ is large where the slope is steep and small near the bottom, the steps shrink by themselves as we get closer: the tangent gives the direction and the magnitude of the step.

The learning rate is the one choice gradient descent leaves to you, and it matters. Play with it on the simplest valley, $f(x) = x^2$:

A too large $\eta$ jumps over the minimum, and can even diverge. A too small $\eta$ converges, but needs a large number of steps. For a network, each step costs a forward and a backward pass over the data, so the number of steps is the cost of training: choosing $\eta$ is a real trade-off, and much of practical deep learning is about it.

Gradient descent gets to a local minimum: the bottom of the valley it starts in, not necessarily the deepest one. The animation at the top of this page shows it: the point stops in the first valley it meets.

4. Many knobs at once: the gradient

4.1 Functions of several variables

A loss depends on all the knobs of the network, so we need functions of several variables. A function from $\mathbb{R}^n$ to $\mathbb{R}^m$ takes $n$ numbers and returns $m$:

• $f(x, y) = x^2 + y^2$, from $\mathbb{R}^2$ to $\mathbb{R}$;
• $f(x, y, z) = x + y + z$, from $\mathbb{R}^3$ to $\mathbb{R}$;
• $f(x, y) = \begin{pmatrix} x + y \\ x - y \end{pmatrix}$, from $\mathbb{R}^2$ to $\mathbb{R}^2$: each output, $f_1(x, y) = x + y$ and $f_2(x, y) = x - y$, is a function of all the inputs.

The names of the variables don’t matter: $f(x, y)$, $f(x_1, x_2)$ and $f(w_1, w_2)$ are the same kind of function, and $f(\mathbf{x})$, with $\mathbf{x}$ in bold, means that the input is a vector. From now on we focus on functions from $\mathbb{R}^n$ to $\mathbb{R}$: many numbers in, a single number out, like a loss.

4.2 Partial derivatives

The partial derivative $\dfrac{\partial f}{\partial x_j}$ is the derivative of $f$ with respect to $x_j$, the other variables being fixed: turn the knob $x_j$ only, and watch $f$. In practice, treat every other variable as a constant and differentiate as usual, term by term:

and in the same way $\frac{\partial f}{\partial y} = x + z + 2y$ and $\frac{\partial f}{\partial z} = x + y + 2z$. Read $\frac{\partial f}{\partial y}$ as how much $f$ changes when $y$ changes.

The chain rule works with partial derivatives too. With $\hat y = x_1 - x_2$ and $\mathcal{L} = (\hat y - y)^2$:

4.3 The gradient

Put all the partial derivatives in one vector, and you get the gradient:

One entry per variable, so a vector of size $n$. The gradient is itself a function, from $\mathbb{R}^n$ to $\mathbb{R}^n$: at each point $\mathbf{x}$, it gives the vector $\nabla f(\mathbf{x})$, which points in the direction where $f$ increases the fastest. For the bowl $f(x_1, x_2) = x_1^2 + x_2^2$, $\nabla f = (2x_1, 2x_2)$ points away from the bottom, uphill, everywhere.

Gradient descent in several variables is the same rule as in section 3, with vectors. Each step goes against the gradient:

On the bowl $\mathcal{L}(\theta_1, \theta_2) = \theta_1^2 + \theta_2^2$, every step points straight at the centre, and the steps converge to $\theta_\infty = (0, 0)$, the minimum.

5. Training a network

5.1 The training loop

Everything is in place. Put all the weights and all the biases of the network in one vector, $\theta$, and train it by gradient descent on the loss:

where the gradient, one partial derivative per knob, is computed by backpropagation: the chain rule on all the layers. One step of training is always the same three moves:

1. Forward: compute the predictions and the loss.
2. Backward: compute $\nabla \mathcal{L}$, the derivative of the loss with respect to every parameter.
3. Step: move every parameter against its derivative, $\theta \leftarrow \theta - \eta \nabla \mathcal{L}$.

Repeat until the loss stops going down.

5.2 Watch a network train

The Space below is a network with one hidden layer, learning a function from examples. Each step of gradient descent moves every weight and every bias by $-\eta\, \partial \mathcal{L} / \partial \theta$: watch the curve fit the points, and the loss go down.

5.3 Train one by hand

Training a network by hand is long, but training the smallest model is not. Take two examples, $(x, y) = (0, 1)$ and $(1, 3)$, and the model $\hat y = a\,x + b$: a straight line, with two knobs, $a$ and $b$. The loss is the sum of the squared errors on the two examples:

For a straight line, we did not need gradient descent: least squares gives the best $(a, b)$ with a formula, $\theta = (X^\top X)^{-1} X^\top y$, a closed-form solution. That is exactly what a network does not have, and why the whole of section 3 was needed.

6. Recap

• The forward pass computes the prediction layer by layer, $h = \sigma(W x + b)$: matrices, then an activation. With enough neurons, a network can draw almost any shape.
• The derivative is the slope of the tangent: how much a function changes when its input changes. Its sign says which way is up.
• The chain rule differentiates a chain of functions, one factor per layer. Applied from the loss back to every weight, it is backpropagation: one forward pass, one backward pass.
• Gradient descent walks downhill, $x_{n+1} = x_n - \eta f'(x_n)$, to a local minimum. The learning rate $\eta$ must be neither too large (oscillations, divergence) nor too small (too many steps).
• A partial derivative turns one knob with the others fixed; the gradient collects them in a vector that points uphill.
• Training a network is gradient descent on the loss, with backpropagation to compute the gradient: $\theta_{k+1} = \theta_k - \eta \nabla \mathcal{L}(\theta_k)$.