Activation functions compared: ReLU, Sigmoid and Tanh
Activation functions introduce non-linear properties to neural networks, allowing them to learn complex patterns rather than just computing linear transformations. ReLU is the default choice for hidden layers because it avoids the vanishing gradient problem and computes efficiently, while Sigmoid and Tanh are typically reserved for output layers or specific recurrent architectures where bounded outputs are required. Understanding the distinct mathematical properties and limitations of each function is essential for designing effective neural networks and diagnosing training failures.
When you sit down for a machine learning exam, you will rarely be asked simply to define an activation function. University examiners expect you to understand the mathematical consequences of these functions during the backpropagation process. They want to see that you understand how a function’s derivative affects the gradients flowing backwards through a deep architecture. Memorising the formulas is only the starting point. The real marks are awarded for demonstrating the judgement to select the correct function for a specific layer and justifying that choice using the principles of calculus and gradient descent.
Why neural networks need activation functions
To understand why we compare these functions, we must first establish exactly what they do. A standard artificial neuron performs a very simple operation. It takes a set of inputs, multiplies them by a set of weights, adds a bias term, and produces a single number. This is a linear operation. If you stack multiple layers of these neurons on top of one another without an activation function, the entire network collapses mathematically into a single linear transformation.
No matter how many hidden layers you add, a purely linear network can only ever draw a straight line or a flat plane through your data. It would be completely incapable of learning the complex boundaries required for image recognition, natural language processing, or any non-trivial classification task. This limitation is tied to the fact that the composition of multiple linear functions results in just another linear function.
Activation functions solve this problem by applying a non-linear mathematical operation to the output of each neuron before passing the signal to the next layer. By bending or restricting the output space, they allow the neural network to approximate highly complex, continuous functions. This principle is famously formalised in the Universal Approximation Theorem. When you evaluate ReLU, Sigmoid, and Tanh, you are evaluating exactly how they apply this non-linearity and what side effects they introduce to the network’s learning process.
Sigmoid: The classic probability mapper
The Sigmoid function, sometimes called the logistic function, takes any real-valued number and squashes it into a range strictly between 0 and 1. The formula is f(x) = 1 / (1 + exp(-x)). Historically, this was the default activation function for all layers in early neural networks because its S-shaped curve mimics the biological behaviour of a neuron gradually firing. Today, its use in hidden layers is considered obsolete due to severe mathematical drawbacks during training.
The primary issue with Sigmoid is the vanishing gradient problem. To update the weights of a neural network, we use backpropagation, which relies heavily on the chain rule of calculus. This requires us to calculate the derivative of the activation function. The derivative of the Sigmoid function reaches a maximum value of 0.25 when the input is exactly zero, and it approaches zero as the input moves towards positive or negative infinity.
When you multiply these small derivatives together across multiple hidden layers, the error gradients shrink exponentially. By the time the gradient signal reaches the earliest layers of a deep network, it is so infinitesimally small that the weights barely update. The early layers simply stop learning. Because of this fatal flaw, Sigmoid is now almost exclusively used in the final output layer of binary classification models. In that specific context, its ability to compress any raw logit into a clean probability distribution between 0 and 1 is highly desirable.
Tanh: The zero-centred alternative
The Hyperbolic Tangent function, or Tanh, is mathematically very similar to Sigmoid. In fact, it is simply a scaled and shifted version of the Sigmoid function. It squashes inputs into a range between -1 and 1. The formula is f(x) = (exp(x) - exp(-x)) / (exp(x) + exp(-x)). Because it crosses the origin, Tanh provides a zero-centred output, which gives it a distinct advantage over Sigmoid when training deep networks.
When an activation function is not zero-centred, the outputs sent to the next layer are always positive. During backpropagation, this means the gradients of the weights will either be all positive or all negative for a given layer. This restriction causes the weight updates to occur in a zigzagging fashion, making gradient descent highly inefficient and slowing down convergence. By outputting values between -1 and 1, Tanh ensures that the mean of the activations is closer to zero. This balances the weight updates and allows the network to converge much faster.
Despite this improvement in convergence speed, Tanh still suffers from the vanishing gradient problem. Like Sigmoid, the function flattens out at the extremes. If a neuron receives a large positive or negative input, the output of Tanh approaches 1 or -1, and the local derivative approaches zero. If the derivative is zero, no gradient flows backwards, and learning halts. Consequently, while Tanh is generally preferred over Sigmoid for hidden layers, it is still mostly restricted to specific architectures like Recurrent Neural Networks where bounded outputs are strictly necessary to prevent exploding activations across time steps.
ReLU: The modern default for hidden layers
The Rectified Linear Unit, commonly known as ReLU, is currently the most widely used activation function for hidden layers in deep learning. Its formula is incredibly simple: f(x) = max(0, x). If the input is negative, the function outputs zero. If the input is positive, the function outputs the input exactly as it is. It operates on a range from 0 to positive infinity.
The genius of ReLU lies in its derivative. For any positive input, the derivative is exactly 1. For any negative input, the derivative is exactly 0. Because the positive derivative is exactly 1, the error gradient does not shrink as it is multiplied backwards through the layers during backpropagation. The error signal passes through perfectly, effectively solving the vanishing gradient problem that plagued Sigmoid and Tanh. This single mathematical property is what enabled researchers to successfully train networks with dozens or even hundreds of hidden layers.
However, ReLU introduces a unique vulnerability known as the dying ReLU problem. Because the derivative is exactly zero for all negative inputs, a neuron can become completely inactive. As a worked example, imagine a neuron with a pre-activation output of -3.0. The ReLU function turns this into 0. During backpropagation, the local gradient is 0. Any error signal passed backwards is multiplied by 0, meaning the weights feeding into this neuron will not update. If a high learning rate causes a massive weight update that pushes the pre-activation value into the negative range for all training examples, the neuron will never fire again. It is permanently dead.
Despite this flaw, ReLU remains the industry standard. It is highly computationally efficient because it only requires checking if a number is greater than zero, avoiding the expensive exponential calculations required by Sigmoid and Tanh.
How to answer this in an exam
When an exam question asks you to compare activation functions, a mark scheme will heavily reward the mathematical justification behind your comparison. Do not simply draw the shapes of the curves and state the output ranges. You must explicitly discuss the derivatives of these functions and how those derivatives interact with the chain rule during backpropagation.
If you are asked to design an architecture, clearly state that you are placing ReLU in the hidden layers to prevent vanishing gradients. If the problem is a binary classification task, state that you are using Sigmoid in the final output layer to map the unbounded logits to a valid probability. Always link the structural choice back to the mathematical behaviour of the gradient. Our Full Marks Press Neural Networks guide covers this exact topic with worked exam questions, showing you how to structure a full-mark essay response on backpropagation dynamics and architectural choices.
FAQ
Why is ReLU considered non-linear if it looks like a straight line? ReLU is piecewise linear. While it consists of two linear segments, the sharp bend at zero makes the function as a whole non-linear. This bend is sufficient to break the linearity of the network, allowing multiple stacked ReLU layers to approximate highly complex, curved decision boundaries.
Can I use ReLU in the output layer of a neural network? You should only use ReLU in an output layer if you are performing a regression task where the target variable is strictly a positive continuous number, such as predicting the price of a house. For classification tasks, ReLU is entirely inappropriate for the output layer because it does not map values to a probability distribution.
What is Leaky ReLU and why was it invented? Leaky ReLU is a variation of the standard ReLU function designed to fix the dying ReLU problem. Instead of outputting exactly zero for negative inputs, it introduces a very small slope, such as f(x) = 0.01x for negative values. This ensures the derivative is never strictly zero, allowing dead neurons a chance to recover during training.
Which activation function should I state if a question lacks specific context? If an exam question asks for a default activation function for a hidden layer without providing specific architectural constraints, always state ReLU. It is the universally accepted standard for feedforward neural networks and convolutional neural networks due to its computational efficiency and gradient preservation.
For more detailed breakdowns of exam strategies and complete mark scheme analyses, you can download a free copy of our study resources at fullmarkspress.com/free.