What is backpropagation and how do you calculate it in an exam?
Backpropagation is the algorithm used to train neural networks by calculating the gradient of the loss function with respect to each individual weight and bias. It works backwards from the network output layer to the input layer, applying the chain rule of calculus to determine precisely how much each parameter contributed to the overall prediction error. You calculate it in an exam by first mapping out the computational graph of the network, computing the local gradients for each node, and multiplying them together from right to left.
Understanding backpropagation is arguably the most critical requirement for passing any university machine learning module. When a network makes a prediction, the forward pass computes the mathematical error. Backpropagation is the mechanism that tells the network how to adjust its parameters to reduce that error on the next iteration. In an exam setting, examiners are testing your understanding of calculus and computational graphs rather than your ability to memorise abstract definitions. You need to show that you can trace a mathematical path through a network architecture, managing intermediate values and partial derivatives without getting lost in the notation.
The Core Mechanics of Forward and Backward Passes
Before you can calculate backpropagation, you must fully understand the forward pass. During the forward pass, numerical data moves through the neural network from inputs to outputs, applying weights, biases, and activation functions at each mathematical step. This process terminates at the loss function, which quantifies the difference between the network prediction and the actual ground truth label. You cannot perform a backward pass without the intermediate values calculated during this forward phase. These cached values are essential for computing the local gradients later, and failing to write them down is a common way students lose marks.
Once the final loss is established, the backward pass begins. The goal is to find the partial derivative of the loss with respect to every single weight and bias in the network architecture. This partial derivative represents the slope of the loss function. If the computed derivative is positive, increasing the weight will increase the loss, so the optimiser knows to decrease the weight. If the derivative is negative, the optimiser will increase the weight to reach a lower error.
The backward pass calculates these derivatives efficiently by reusing computations rather than starting from scratch for every parameter. Instead of calculating the gradient for an early layer independently, backpropagation takes the gradient calculated at the subsequent layer and multiplies it by the local gradient of the current layer. This recursive approach saves immense computational power in practice and provides a logical, structured sequence that examiners want to see you demonstrate on paper.
The Chain Rule as Your Mathematical Engine
At its absolute core, backpropagation is just the chain rule of calculus applied repeatedly across a graph of operations. If you have a variable z that depends on y, and y depends on x, the chain rule states that the derivative of z with respect to x is the product of the derivative of z with respect to y and the derivative of y with respect to x. In the context of neural networks, z represents the final scalar loss, and x represents a specific weight you want to update.
In an exam paper, you will likely be given a small, simplified network or a specific computational graph to evaluate. Each node in this graph represents a mathematical operation, such as addition, multiplication, or a non-linear activation like a Sigmoid or ReLU function. To apply the chain rule correctly, you must calculate the local gradient at each node. The local gradient is simply the derivative of a node output with respect to its immediate inputs, temporarily ignoring the rest of the network.
The global gradient, which is the final value you actually need for the weight update, is then found by taking the gradient passed backwards from the node ahead of it and multiplying it by the local gradient. You can think of this as upstream gradient multiplied by local gradient equals downstream gradient. Memorising this exact sequence prevents you from getting lost in complicated partial derivative notation and keeps your exam working methodical and transparent for the marker.
A Step-by-Step Worked Example
Let us walk through a typical exam scenario to see this in practice. Imagine a very simple computational graph where an input x is multiplied by a weight w to produce an intermediate value z. Then, z is passed through a ReLU activation function to produce the prediction y. Finally, we use a squared error loss function defined as L = (y − t)2, where t is the target value. Assume the exam question sets the initial values as x = 2, w = −1, and t = 1.
Step one is the forward pass. We calculate z = x × w, which is 2 × −1 = −2. Next, we apply the ReLU function, which outputs the maximum of 0 and z. Since z is −2, the prediction y becomes 0. Finally, we calculate the loss L = (0 − 1)2, which equals 1. You must write these steps down clearly on your exam script. Examiners often award method marks for a correct forward pass even if you make a silly arithmetic error later in the backward pass. Keep your intermediate values visible.
Step two is the backward pass, starting at the loss function and moving backwards to w. First, find the derivative of the loss with respect to y. Using the standard power rule, the derivative is 2(y − t). Plugging in our values gives 2(0 − 1) = −2. This −2 is our initial upstream gradient. Now, we move back to the ReLU node. The derivative of ReLU is 1 if the input is greater than 0, and 0 if the input is less than or equal to 0. Since our cached input z was −2, the local gradient is 0.
Finally, we apply the chain rule to find the derivative of the loss with respect to the weight. We multiply our gradients together. We know the upstream part is −2 and the local gradient of the activation is 0. Because this upstream gradient encounters a zero, the final gradient for w is also 0. The weight will not be updated by the optimiser. This specific phenomenon is called a dead ReLU, and adding a brief sentence pointing out why the gradient became zero will often secure you top marks in a university paper by showing high-level conceptual understanding.
Handling Matrix Calculus and Multiple Dimensions
While scalar examples are excellent for building intuition, university machine learning modules frequently test backpropagation using vectors and matrices. When inputs and weights are multidimensional, the fundamental chain rule principles remain exactly the same, but the standard scalar multiplication turns into dot products and matrix multiplications. The local gradients become Jacobian matrices, which contain the partial derivatives of all outputs with respect to all inputs.
A frequent trap in exams is getting the matrix dimensions wrong during the backward pass. The golden rule for matrix backpropagation is that the gradient of the loss with respect to a parameter must have the exact same dimensions as the parameter itself. If your weight matrix W is a 3 by 2 matrix, then the gradient of the loss with respect to W must also be a 3 by 2 matrix. If you end up with mismatched dimensions, you have likely multiplied your matrices in the wrong order or forgotten to transpose a specific matrix.
To avoid this trap in a timed exam, write out the shapes of every vector and matrix underneath your equations before you calculate the numbers. For example, if you know the upstream gradient is a 3 by 1 vector and the local input is a 2 by 1 vector, and you need a 3 by 2 matrix for the final weight gradient, you immediately know you must compute the outer product of the upstream gradient and the transposed input. Tracking shapes is a highly reliable diagnostic tool that highlights algebraic errors before they ruin your entire calculation.
How to answer this in an exam
Mark schemes for backpropagation questions are highly predictable across UK universities. Examiners are looking for a systematic approach, not just the final numerical answer. Begin by explicitly defining the computational graph or the chain rule expansion. Write out the full formula representing the partial derivatives before you plug in any numbers. This secures your foundational marks and proves you know the algorithm.
Next, structure your answer by clearly separating the forward pass from the backward pass. List your intermediate node values distinctly. When calculating local gradients, state the derivative rule you are applying, such as the derivative of the Sigmoid function. This proves you understand the underlying calculus rather than just copying a memorised process. The Full Marks Press Deep Learning and Generative AI guide covers this with worked exam questions, showing exactly how examiners allocate points for each step of the calculation and where students typically drop marks.
Finally, double-check your arithmetic. Simple multiplication errors are the leading cause of dropped marks in backpropagation questions. If the question asks for a weight update, remember to multiply your final gradient by the given learning rate and subtract it from the original weight. Do not just stop at finding the gradient, as the final marks are reserved for executing the actual parameter update.
Frequently Asked Questions
What is the difference between backpropagation and gradient descent? Backpropagation is the specific algorithm used to compute the gradients of the loss function with respect to the network parameters. Gradient descent is the optimisation algorithm that actually uses those computed gradients to update the parameters and minimise the loss. Backpropagation finds the slope; gradient descent takes the step down the hill.
Why do we need activation functions for backpropagation? Without non-linear activation functions, a neural network is just a series of linear transformations, which mathematically collapses into a single linear model regardless of how many layers you add. Activation functions introduce non-linearity, allowing the network to learn complex patterns, and their specific derivatives serve as crucial multipliers in the backward pass.
What is the vanishing gradient problem? The vanishing gradient problem occurs when local gradients (especially from activation functions like Sigmoid or Tanh) are repeatedly less than one. As these small fractions are multiplied together during the backward pass through many layers, the global gradient becomes infinitesimally small, preventing the early layers of the network from updating and learning.
Do I need to memorise the derivatives of common activation functions for exams? Yes. Unless explicitly provided in the exam paper appendix, you are generally expected to know the derivatives for ReLU, Sigmoid, Tanh, and Softmax. Memorising how these derivatives are expressed in terms of their own outputs (for example, the derivative of a Sigmoid output y is y(1 − y)) will save you valuable time during the exam.
For a complete breakdown of exam strategies across all your machine learning modules, you can download a free copy of our revision materials at fullmarkspress.com/free.