How to deploy a machine learning model: a student’s guide
Deploying a machine learning model means taking a trained algorithm and making its predictions available to users or other software systems in a production environment. The standard process involves saving the trained model artifact, packaging it with its software dependencies into a container, and exposing it via an API endpoint on a cloud server.
In your university coursework, you spend most of your time cleaning data and tuning hyperparameters in local environments. However, a mathematical model sitting on a laptop provides zero value to a business. As a practitioner, I spend far more time moving models from that experimental phase into a live, highly available environment. Exam boards are highly aware of this industry reality, which is why modern machine learning modules increasingly test your understanding of deployment architectures. You need to demonstrate that you can bridge the gap between a static Python object and a functioning software service. This guide breaks down the standard deployment pipeline so you can explain it clearly under exam conditions.
Packaging the model with containers
Once a model finishes training, you must save it as a file, commonly referred to as an artifact. Formats like pickle for scikit-learn or SavedModel for TensorFlow capture the mathematical weights and the architecture of your network. But an artifact alone cannot execute predictions. It requires specific versions of Python, numerical libraries, and operating system packages to function correctly. If you attempt to run it on a different machine without these exact dependencies installed, the code will crash.
To solve this environment problem, the industry standard is containerisation, almost exclusively using Docker. A Docker container acts like a lightweight, isolated computer that holds your model artifact, the inference script, and a text file listing every single dependency. When you build the container image, the software downloads the exact libraries required. This guarantees that the model will behave exactly the same way on a developer’s laptop as it does on a massive cloud server.
Consider a worked example of a student building a sentiment analysis tool. Instead of sending the raw PyTorch files to the engineering team, you write a Dockerfile. This file starts from a base Python image, installs PyTorch version 2.1, copies the trained weights, and runs a web server script. In an exam, if you are asked how to ensure consistency between development and production, containerisation is your definitive answer. Examiners want to see you explicitly mention Docker and the isolation of dependencies.
Choosing a deployment pattern
Not all models deliver predictions in the same way. The two primary deployment patterns you must understand for your exams are batch prediction and real-time inference. Batch prediction involves running a large volume of data through the model all at once, usually on a scheduled basis. For example, a retail bank might run a credit-scoring model every night at 2:00 AM to evaluate millions of customer accounts. The predictions are saved to a database, ready for analysts to view the next morning. Batch deployment is computationally efficient and does not require immediate responses.
Real-time inference requires the model to generate a prediction the moment a request arrives. Imagine an e-commerce website recommending products as a user clicks through different pages. The model must process the user’s current action and return a recommendation in milliseconds. To achieve this, the containerised model is wrapped in a web framework like FastAPI or Flask. This framework exposes a REST API or a gRPC endpoint that other software systems can call.
When an exam question presents a scenario and asks you to design a deployment strategy, you must justify your judgement between these two patterns. Look for clues in the prompt. If the scenario demands immediate action, such as detecting fraudulent credit card transactions at the point of sale, you must specify a real-time API deployment. If the prompt describes generating weekly sales forecasts, recommend a batch process. Demonstrating this architectural judgement shows the examiner you understand the practical constraints of software engineering.
Serving and scaling the infrastructure
Once you choose real-time inference and build your container, you need a place to run it. A single container on a single cloud server is a highly vulnerable deployment strategy. If that specific server crashes, your model goes offline and the dependent applications fail. In a production setting, engineering teams use container orchestration tools, most notably Kubernetes. Kubernetes manages clusters of servers and ensures that your model remains available even during hardware failures.
Scaling is a critical concept to grasp for your module exams. If your sentiment analysis API suddenly goes viral and receives thousands of requests per second, a single container will be overwhelmed. The requests will queue up, latency will spike, and the system might crash. Kubernetes handles this through a process called horizontal scaling. It monitors the CPU and memory usage of your model containers. When the load increases, it automatically spins up duplicate copies of your container to share the incoming traffic.
A load balancer sits in front of these duplicate containers. When a user sends an HTTP request to the API endpoint, the load balancer receives it and routes it to the container with the most available capacity. In your exam answers, detailing this flow from user request to load balancer, then to a scaled Kubernetes pod, and finally returning the prediction via HTTP proves you understand enterprise deployment. It elevates your answer from a basic description to a comprehensive systems architecture response.
Monitoring for drift and performance
Deployment is not the final step of the machine learning lifecycle. Once a model is live, its predictive performance will inevitably degrade over time. The real world changes constantly, but the model remains frozen on the historical data it learned from during the training phase. This phenomenon is known as drift, and examiners frequently test your ability to diagnose and mitigate it. You must establish continuous monitoring pipelines to track both software metrics and statistical metrics.
Software monitoring tracks the health of the API itself. You need to log inference latency, error rates, and memory consumption. If your model takes too long to return a prediction, it might cause a timeout error in the main user application. Statistical monitoring, however, is entirely unique to machine learning. You must watch for data drift, which occurs when the distribution of the incoming production data shifts away from the training data. For example, if you trained a housing price model during an economic boom, the input features will look very different during a recession.
You must also monitor concept drift, which happens when the fundamental relationship between the input features and the target variable changes. To detect this, you eventually need to collect ground truth labels for your production predictions and compare them. If the accuracy drops below a predefined threshold, the monitoring system triggers an alert, prompting the data science team to retrain the model on fresh data. Mentioning the strict distinction between data drift and concept drift is a highly effective way to secure top marks.
How to answer this in an exam
When faced with an exam question about deployment, the mark scheme heavily rewards candidates who frame their answers around architectural trade-offs. Do not simply list cloud providers or software tools. Instead, explain exactly why a specific tool or architectural pattern is necessary for the scenario provided. If the question asks how to deploy a recommendation engine, state clearly that you would package the model artifact as a Docker image to guarantee environment consistency across different deployment stages.
Furthermore, examiners look for an understanding of the full lifecycle. A common pitfall for university students is stopping the explanation the moment the API endpoint goes live. You must explicitly mention how the system handles peak computational loads through horizontal scaling and how it guards against degradation through continuous monitoring. Use precise terminology throughout your essay. Words like RESTful API, inference latency, data drift, and container orchestration demonstrate practitioner-level knowledge.
For instance, a strong exam answer will argue that while batch processing is cheaper and easier to maintain, a fraud detection system absolutely requires the low latency of a real-time REST API, despite the higher infrastructure costs. The Full Marks Press MLOps guide covers this with worked exam questions, showing you exactly how to structure these justifications to hit every point on the grading rubric.
Frequently asked questions
What is the difference between an API and a web application? An API is a software intermediary that allows two applications to talk to each other, returning raw data formats like JSON. A web application includes a graphical user interface designed for human interaction. Deployed machine learning models typically use APIs so other software systems can consume their predictions programmatically.
Why cannot I just run my Jupyter Notebook in production? Jupyter Notebooks are excellent for data exploration but lack the rigid structure required for production software. They do not handle concurrent user requests well, are notoriously difficult to version control, and lack automated testing mechanisms. Production systems require stateless and scalable Python scripts.
What is a model registry? A model registry is a centralised repository that stores and versions trained machine learning models. It acts similarly to Git for your model artifacts, tracking exactly which version is currently deployed in production. This allows you to roll back immediately to a previous version if a newly deployed model underperforms.
How do I test a deployed model before routing all traffic to it? Engineers use a technique called shadow deployment. In a shadow deployment, the new model receives a copy of live production traffic, but its predictions are only logged to a database and not returned to the user. This allows you to verify its latency and accuracy safely before making it the primary production model.
For more detailed examples and practice scenarios, download a free copy of our revision materials at fullmarkspress.com/free.