Image segmentation techniques: a revision guide for computer science students

Image segmentation is the computer vision process of partitioning a digital image into multiple distinct regions by grouping pixels that share similar attributes. Instead of assigning a single label to an entire image, segmentation assigns a specific class or object identifier to every individual pixel. This pixel-level classification simplifies the image representation, making it easier for machine learning models to analyse specific objects and their precise boundaries.

When you look at a photograph of a street, your visual cortex instantly separates cars, pedestrians, buildings, and the sky. Image segmentation algorithms attempt to replicate this biological capability mathematically. For university assessments, you need to understand both the traditional methods that rely on pixel intensity values and the modern deep learning architectures that extract complex spatial hierarchies. Examiners want to see that you can distinguish between different types of segmentation tasks and that you know exactly how to measure a model’s success. This guide will walk you through the core concepts you need to memorise and apply, bridging the gap between high-level theory and the specific technical details required to secure top marks.

The three primary types of segmentation

Semantic segmentation treats multiple objects of the same class as a single entity. If an image contains three dogs, semantic segmentation labels every pixel belonging to any dog as the “dog” class. It makes no distinction between the first, second, or third dog. This approach is highly effective for tasks like land cover classification in satellite imagery, where you only care about identifying all areas of forest versus all areas of water. In an exam, if a question asks you to map drivable road surfaces for an autonomous vehicle, semantic segmentation is your most computationally efficient answer.

Instance segmentation identifies and separates individual occurrences of objects within the same class. Returning to the previous example, instance segmentation will label the pixels of the first dog as one instance, the second as a second instance, and the third as a third. This technique is essential when counting objects or tracking specific items across video frames, such as tracking individual blood cells in medical imaging. However, instance segmentation typically only focuses on countable objects and often ignores amorphous background elements like the sky or the floor.

Panoptic segmentation unifies both semantic and instance approaches into a single model output. It assigns two labels to every pixel: a semantic class and an instance identifier. For amorphous background regions, known as “stuff” categories, the instance identifier is simply ignored. For countable objects, known as “thing” categories, the instance identifier differentiates each unique item. While computationally heavy, panoptic segmentation provides a complete mathematical understanding of a visual scene and is becoming increasingly common in advanced robotics modules.

Traditional techniques based on pixel intensity

Before neural networks dominated computer vision, engineers relied on traditional mathematical techniques to segment images. Thresholding is the most fundamental of these methods. It works by converting a greyscale image into a binary image based on a specific pixel intensity value. If a pixel’s intensity is greater than the threshold, it is assigned to the foreground; if it is lower, it becomes part of the background. Otsu’s method is the classic algorithm you should quote in an exam for automatic thresholding. It calculates the optimum threshold value by minimising the variance within the foreground and background pixel classes, removing the need for manual guessing.

Region growing offers a slightly more sophisticated traditional approach. The algorithm starts with a small group of seed pixels selected either manually or automatically. It then examines neighbouring pixels and adds them to the seed region if they meet a specific similarity criterion, such as having a colour or intensity value within a defined tolerance. The region continues to grow outwards until no more similar pixels are found.

While you might think these traditional methods are outdated, examiners frequently test them to ensure you understand the foundational mathematics of computer vision. A common exam scenario might ask you to segment bright white tumours against a dark MRI background. In such a highly controlled, high-contrast environment, simple thresholding is often a more appropriate and less computationally expensive solution than deploying a massive deep learning model.

Deep learning architectures for pixel-level prediction

Modern image segmentation relies heavily on convolutional neural networks, but standard architectures are designed for image classification and output a single probability distribution for the whole image. To achieve segmentation, researchers developed Fully Convolutional Networks. This architecture replaces the final dense layers of a standard network with transposed convolutional layers. This structural change allows the network to upsample the compressed feature maps back to the original image dimensions, producing a dense prediction matrix where every pixel receives a class probability.

The most famous architecture you will encounter in your revision is the U-Net. Originally developed for biomedical image segmentation, U-Net features a symmetrical encoder-decoder structure that resembles the letter U. The encoder path gradually reduces the spatial dimensions of the image while increasing the number of feature channels to capture the overall context of the scene. The decoder path then expands these spatial dimensions back to the original resolution to achieve precise localisation.

The defining feature of U-Net, and the element you must mention in any exam answer, is its use of skip connections. As the image is compressed during encoding, fine-grained spatial information is lost. Skip connections bypass the deepest layers of the network, directly concatenating the high-resolution feature maps from the encoder with the upsampled feature maps in the decoder. This allows the model to combine high-level contextual information with low-level spatial detail. If you want to see exactly how U-Net questions are structured in university assessments, the Full Marks Press Computer Vision guide covers this with worked exam questions, showing you how to draw and label the architecture correctly.

Evaluating segmentation models mathematically

Writing out an architecture is only half the battle; you must also prove to the examiner that you know how to measure its accuracy. Standard classification metrics like overall accuracy are misleading in segmentation tasks due to class imbalance. Imagine a medical scan where a tumour occupies just one per cent of the image. A model that completely fails to find the tumour, but correctly identifies the 99 per cent background, will still achieve 99 per cent accuracy. This is why we use specialised overlap metrics.

Intersection over Union, also known as the Jaccard Index, is the most common metric. It calculates the area of overlap between the predicted segmentation mask and the ground truth mask, divided by the total area encompassed by both masks combined. The formula is simply the intersection divided by the union. A perfect prediction yields a score of 1.0, while a complete failure yields 0.0.

The Dice Coefficient is another crucial metric, particularly in medical imaging modules. It is mathematically very similar to Intersection over Union but places twice the weight on the intersecting area. The formula is two times the intersection, divided by the sum of the total pixels in the prediction and the total pixels in the ground truth.

Let us look at a short worked example. Suppose you have a ground truth mask containing 100 pixels of a car. Your model predicts a mask of 120 pixels. The intersecting area, where both the ground truth and the prediction agree, is 80 pixels. To find the Intersection over Union, you divide the intersection (80) by the union (100 plus 120 minus the 80 overlapping pixels, which equals 140). This gives an Intersection over Union of 0.57. For the Dice Coefficient, you multiply the intersection by two (160) and divide by the sum of the individual areas (100 plus 120 equals 220), resulting in a Dice score of 0.72. Showing this calculation in an exam proves you understand the mechanics behind the metrics.

How to answer this in an exam

When examiners mark a question on image segmentation, they are actively looking for your ability to justify your technical choices. A weak answer simply lists the algorithms. A top-tier answer explains why a specific algorithm is the most suitable for the scenario provided in the prompt. If the question involves real-time processing on a low-power device, such as a drone, the mark scheme will reward you for suggesting lightweight semantic segmentation models and penalise you for suggesting computationally massive panoptic networks.

Furthermore, you must be precise with your terminology. Never confuse bounding boxes with segmentation masks. Object detection draws a rectangular box around an item, whereas segmentation traces the exact pixel boundary. Mark schemes frequently deduct points from students who use detection terminology, like “bounding box coordinates”, when discussing segmentation architectures. Always refer to “pixel-level classification”, “segmentation masks”, and “dense predictions”.

Finally, always state the loss function if you are describing a neural network approach. Standard mean squared error is rarely appropriate here. The mark scheme will specifically look for Cross-Entropy Loss for multi-class semantic segmentation, or Dice Loss when dealing with heavily imbalanced datasets like medical scans. Mentioning the correct loss function demonstrates a complete, end-to-end understanding of how the model is trained, elevating your answer from a basic description to a comprehensive engineering solution.

Frequently asked questions

Why is overall accuracy a bad metric for image segmentation? Overall pixel accuracy is heavily skewed by class imbalance. If the background makes up the vast majority of an image, a model can achieve a very high accuracy score simply by predicting the background class for every pixel, completely failing the actual segmentation task.

What is the difference between image classification and image segmentation? Image classification assigns a single label to an entire image, such as labelling a whole photograph as a cat. Image segmentation assigns a class label to every individual pixel within that photograph, detailing exactly where the cat is located and its exact shape.

How do skip connections improve segmentation models? Deep neural networks compress images to extract complex features, losing high-resolution spatial details in the process. Skip connections feed the high-resolution data from the early layers directly into the later layers, allowing the model to reconstruct sharp boundaries around objects.

Can unsupervised clustering algorithms be used for image segmentation? Yes, unsupervised clustering algorithms like K-Means can segment images by grouping pixels with similar colour or intensity values. While useful for simple tasks without labelled training data, they struggle with complex textures and varying lighting conditions compared to neural networks.

Find more practitioner-written revision materials and grab a free copy of our exam technique guides at fullmarkspress.com/free.