What are the common pitfalls when training neural networks?

The most common pitfalls when training neural networks are overfitting to the training data, choosing an inappropriate learning rate, and encountering vanishing or exploding gradients. These issues either prevent the model from converging during the training phase or completely destroy its ability to generalise to unseen data. Recognising these traps early allows you to apply the correct architectural changes or regularisation techniques to stabilise the learning process.

As a university student revising for a machine learning exam, you need to look beyond the code. Examiners want to see that you understand the underlying mathematics and the practical realities of training deep architectures. When a model fails to learn, it rarely crashes with a helpful error message. Instead, it fails silently, producing poor predictions or a stagnant loss curve. Understanding these failure modes separates a competent practitioner from someone who just imports a pre-written library and hopes for the best.

Overfitting and Memorising the Noise

A neural network with millions of parameters has an enormous capacity to learn. If left unchecked, it will not just learn the underlying mathematical patterns in your dataset; it will memorise the statistical noise and the specific quirks of the training examples. You will see the training loss approach zero while the validation loss plateaus and then begins to climb. The model becomes highly specific to the training set and functionally useless in the real world.

Consider a network trained to classify images of cats and dogs. If all the training photos of cats happen to be taken indoors on carpets, and all the dogs are outside on grass, an overfitted network might simply learn to detect green pixels rather than actual animal features. It has memorised the background noise instead of learning the target features. This happens frequently when the dataset is too small relative to the complexity of the neural network you are trying to train.

To fix overfitting, practitioners apply regularisation techniques. Adding an L2 penalty to the loss function forces the network to keep its weights small, preventing any single feature from dominating the final prediction. This encourages the network to distribute its learning across many features rather than relying entirely on one specific input pattern.

Alternatively, dropout layers offer a structural solution. Dropout randomly deactivates a percentage of neurons during each training pass. This forces the network to learn redundant representations and stops neurons from co-adapting too closely to specific training examples. By the time inference occurs, the network is essentially acting as an ensemble of many smaller networks, greatly improving its ability to generalise to new data.

Selecting an Inappropriate Learning Rate

The learning rate dictates the size of the steps the optimiser takes across the loss landscape during gradient descent. If you set this value too high, the optimiser will overshoot the global minimum. The loss will oscillate wildly or even diverge, increasing with every single epoch until the weights break down. Conversely, a learning rate that is too low means the network learns at a glacial pace, wasting computational resources and risking getting trapped in a shallow local minimum.

Imagine you are trying to find the lowest point in a dark valley while blindfolded. Your only guide is the slope of the ground under your feet. If you take massive leaps, representing a high learning rate, you might jump right over the bottom of the valley and land higher up on the opposite hill. If you take tiny, millimetre-long steps, representing a low learning rate, it will take you months to reach the bottom. You might even get stuck in a small ditch along the way, falsely thinking you have found the absolute lowest point.

In practice, finding the perfect static learning rate is nearly impossible. Different layers of the network often require different step sizes to learn optimally. Relying on a single fixed number for the entire training run is a common beginner mistake that leads to suboptimal model performance.

Instead, you should apply learning rate schedulers or adaptive optimisers. Schedulers gradually decay the learning rate as training progresses, allowing for fast initial learning followed by careful fine-tuning near the minimum. Adaptive optimisers, such as Adam, automatically adjust the learning rate for each individual parameter based on the historical gradients. This offers a highly practical solution to the learning rate trap and is generally the default choice for modern deep learning tasks.

Vanishing and Exploding Gradients

Deep neural networks rely on backpropagation to update their weights, a process mathematically dependent on the chain rule from calculus. As the error signal travels backwards from the output layer to the early layers, it is multiplied by the derivative of the activation function at each step. If these derivatives are consistently smaller than one, the gradient shrinks exponentially until it vanishes. The early layers stop learning entirely, meaning the foundation of your network remains essentially random.

The classic sigmoid activation function is a notorious culprit for vanishing gradients. Because the derivative of a sigmoid function maxes out at exactly 0.25, multiplying several of these together in a deep network reduces the gradient to near zero by the time it reaches the first hidden layer. Exploding gradients represent the exact opposite problem. If the weights are initialised too large, the gradients compound and grow exponentially. This causes numerical overflow, ultimately updating the weights with useless NaN (Not a Number) values and ruining the model.

You can prevent these issues through careful architectural choices. Swapping sigmoid functions for ReLU (Rectified Linear Unit) activations solves the vanishing gradient problem because ReLU has a constant derivative of one for all positive inputs. The error signal can flow backwards through the network without being compressed at every layer.

For exploding gradients, implementing gradient clipping caps the maximum value of the gradient during backpropagation, keeping the updates stable. Additionally, using proper weight initialisation strategies ensures the variance of the outputs remains consistent across layers. Techniques like He initialisation for ReLU networks or Glorot initialisation for tanh networks carefully scale the initial random weights based on the number of inputs and outputs of each layer.

Poor Data Preparation and Scaling

Neural networks expect their input features to be on a similar numerical scale. If you feed raw, unscaled data into a network, the loss landscape becomes highly distorted, resembling an elongated, narrow bowl. The optimiser will struggle to navigate this uneven terrain, leading to painfully slow convergence or a complete failure to learn. Many students overlook data preparation, assuming the neural network will magically figure out the scale eventually.

As a worked example, imagine training a network to predict house prices. Feature A is the number of bedrooms, ranging from 1 to 5. Feature B is the floor area in square feet, ranging from 500 to 5,000. Because the numerical values for Feature B are a thousand times larger, the initial gradients associated with the floor area will completely dominate the weight updates. The network will spend all its time trying to optimise the weights for the floor area while virtually ignoring the number of bedrooms.

You must scale your inputs before training begins to avoid this imbalance. Standardisation subtracts the mean and divides by the standard deviation for each feature. This centers the data around zero with a variance of one. Min-max scaling is another highly effective option, squeezing all values into a uniform range between zero and one. Our Full Marks Press Neural Networks guide covers this exact scenario with worked exam questions, showing you how to calculate and apply these transformations step by step. These scaling techniques ensure a smooth, symmetrical loss landscape, allowing gradient descent to take direct, efficient steps towards the minimum.

How to answer this in an exam

When an exam question asks you to diagnose a training failure, do not just list techniques randomly. Mark schemes heavily reward a structured approach based on diagnosis, theoretical reasoning, and practical application. First, identify the symptom from the prompt. Is the training loss decreasing while validation loss increases? State clearly that this is overfitting. Is the loss fluctuating wildly from epoch to epoch? Point to an excessively high learning rate.

Next, provide the specific mathematical or architectural reason for the failure. Do not just write “use ReLU”. Write that ReLU prevents vanishing gradients because its derivative does not compress positive values during backpropagation. Finally, suggest a concrete intervention. Examiners want to see that you can link the theoretical symptom to a practical cure, demonstrating true engineering judgment. Naming the exact regularisation technique or optimiser required will secure the highest marks.

FAQ

Why does my validation loss suddenly spike during training? This is a classic sign that your learning rate is too high, causing the optimiser to overshoot the minimum, or that the model has begun to overfit the training data. Implement early stopping to halt the training process before this spike permanently ruins your model weights.

What is the difference between batch size and epoch? An epoch is one complete pass through your entire training dataset. The batch size is the number of training examples processed in one forward and backward pass before the network updates its weights. Choosing a batch size that is too large can lead to poor generalisation, while a batch size that is too small can make the training process unstable.

How do I know if my network is too deep? If the training loss does not decrease and you are using sigmoid or tanh activations, your network might be suffering from vanishing gradients due to excessive depth. You might also notice that the weights in the earliest layers remain completely unchanged from their initialised values when you inspect them.

Should I always use dropout? Dropout is highly effective for reducing overfitting in fully connected layers, but it is rarely used in convolutional layers, which naturally have fewer parameters. Only apply dropout if your model shows signs of high variance and is clearly overfitting the training data.

To master these concepts and practice with real past papers, download your free copy at fullmarkspress.com/free.