Overfitting vs underfitting: how to secure top marks in ML assignments
Overfitting occurs when a machine learning model memorises the training data, including its random noise, resulting in poor predictions on unseen data. Underfitting happens when a model is too simple to learn the underlying patterns in the data, leading to inaccurate predictions across both training and unseen datasets. Finding the exact point of balance between these two extremes is the core challenge in training any supervised learning algorithm.
When you sit your university machine learning exam, simply defining these two terms will only secure you the baseline marks. University examiners are looking for applied judgement and practical understanding. They want to see that you can diagnose a poorly performing model from a set of metrics or a learning curve graph. Furthermore, you must demonstrate that you know exactly which mathematical or structural parameters to adjust to fix the issue. Let us break down the specific mechanics of both problems, how to spot them in an exam scenario, and the precise techniques required to resolve them.
Underfitting and the reality of high bias
Underfitting is fundamentally a problem of high bias. In the context of statistical modelling, bias refers to the error introduced by approximating a highly complex real-world problem with an overly simplistic model. The algorithm makes rigid, predetermined assumptions about the shape of the data and consequently fails to capture the true underlying relationship between the input features and the target variable.
Consider a classic exam scenario where you are tasked with predicting house prices based on a wide range of features: square footage, number of bedrooms, distance to the nearest train station, and local crime rates. If you choose a basic linear regression model and only feed it the square footage, you are forcing a strictly straight-line relationship onto a complex, multi-dimensional problem. The model will underfit. It will perform poorly on your training data because a straight line simply lacks the mathematical capacity to map the non-linear realities of the housing market.
Another clear example is attempting to model temperature fluctuations over a calendar year using a linear equation. Temperatures rise in the summer and fall in the winter, forming a curve. A linear model will draw a straight line through the middle of the data, completely missing the seasonal peaks and troughs.
In an exam setting, if a question presents you with a scenario where both the training error and the validation error are exceptionally high, you are looking at an underfit model. The algorithm has not learned enough from the features to make accurate predictions, even on the exact examples it has just been trained on.
Overfitting and the trap of high variance
Overfitting is the exact inverse problem, characterised in the literature by high variance. A model with high variance is overly complex and highly sensitive to the specific fluctuations, outliers, and random noise present in the training set. Instead of learning the general signal that applies to the wider population, the model learns the noise by heart.
A common trap for students in practical coursework assignments is training a highly complex model, such as a deep neural network or a decision tree grown to its absolute maximum depth, on a relatively small dataset. Imagine a decision tree built to predict whether a customer will default on a bank loan. If you do not restrict the depth of the tree, it will keep splitting the data until it creates terminal leaves containing single individuals. It might eventually output a rule deciding that a customer will default because they are exactly 31 years old, live in a specific postal code, and have a first name starting with the letter J.
Similarly, if you fit a 15th-degree polynomial regression model to ten data points, the resulting curve will perfectly intersect every single point in the training set. However, between those points, the curve will swing wildly up and down.
In both cases, the model has achieved near-perfect accuracy on the training data by memorising it. However, when you deploy this model on unseen test data, its predictions will be completely inaccurate. The tell-tale sign of overfitting in an exam question is a near-zero training error paired with a significantly higher validation error. The mathematical gap between these two metrics represents the model’s variance.
Diagnosing the problem using learning curves
To secure top marks in your module, you must be entirely comfortable reading and interpreting learning curves. A learning curve plots the model’s error or loss on the vertical y-axis against the number of training epochs, or the size of the training set, on the horizontal x-axis. Examiners frequently include these graphs in exam papers to test your practical diagnostic skills.
If you are looking at a graph where the training loss and validation loss both decrease initially but then plateau rapidly at a high value, the model is underfitting. The curves will be close together, but the overall error rate will be unacceptable. Adding more training data to this model will not help because the architecture itself lacks the capacity to learn the complexity of the data.
Conversely, if you see the training loss continuing to decrease towards zero while the validation loss stops decreasing and actually begins to curve upwards, you have caught the exact moment the model begins to overfit. As training progresses, the model is memorising the specific noise of the training batch, which actively harms its ability to generalise to the validation batch. The point at the bottom of the validation loss curve, just before it begins to rise, is the optimal stopping point. Everything past that specific epoch is counterproductive.
Proven strategies to correct your model
Once you have successfully diagnosed the problem from the provided metrics or graphs, you must recommend the correct technical solution. Examiners award the majority of marks for matching the right architectural solution to the diagnosed problem.
To fix an underfitting model, you must increase its capacity to learn. You can achieve this by adding more complex features through feature engineering. For example, you might introduce polynomial features or interaction terms to a regression model. Alternatively, you can upgrade the algorithm itself. If a linear model is underfitting, try a non-linear approach like a Support Vector Machine with a Radial Basis Function kernel, or a gradient boosting classifier. Finally, if you have applied heavy regularisation to your model, reducing the regularisation parameter will give the algorithm more freedom to map the data correctly.
To fix an overfitting model, your primary goal is to constrain it. Regularisation is your main tool here. Applying L1 regularisation (Lasso) adds a penalty equal to the absolute value of the magnitude of coefficients, which can force the weights of less important features to exactly zero, effectively performing automatic feature selection. L2 regularisation (Ridge) adds a penalty equal to the square of the magnitude of coefficients, penalising large weights and smoothing out the model’s predictions.
If the exam question specifies you are working with decision trees, you must recommend applying cost-complexity pruning to limit the maximum depth, or setting a minimum number of samples required to split an internal node. For neural networks, implementing dropout layers will randomly deactivate a percentage of neurons during each forward pass, preventing the network from becoming overly reliant on specific activation pathways. Finally, the simplest conceptual cure for overfitting is to provide the model with more diverse training data, making it mathematically harder for the algorithm to memorise individual examples.
How to answer this in an exam
Mark schemes for university machine learning modules are highly predictable when testing this topic. To ensure you drop no marks, you should structure your exam answers using a clear, three-step framework that explicitly targets the marking criteria.
First, define the terms using the correct vocabulary
Frequently Asked Questions
What is the main difference between overfitting and underfitting? Overfitting means the model has learned the training data too closely, including its noise, so it scores well on training data but poorly on new data. Underfitting means the model is too simple to capture the real pattern, so it does poorly on both training and test data.
How do you spot overfitting in an exam answer? Point to a large gap between high training accuracy and low validation accuracy. On learning curves the training error keeps falling while the validation error flattens or rises, and that divergence is the classic sign.
How do you fix overfitting? Get more training data, simplify the model, add regularisation such as L1 or L2, use dropout in a neural network, or stop training earlier once the validation error starts to climb.
How do you fix underfitting? Use a more expressive model, add useful features, reduce the amount of regularisation, or train for longer. Underfitting means the model has not learned enough, so you give it more capacity.