What makes a good AI exam question? A guide for lecturers

A good AI exam question tests a student’s ability to diagnose algorithmic behaviour, justify design choices, and apply theoretical constraints to unseen datasets, rather than simply recalling mathematical formulas. It achieves this by presenting a flawed scenario or a comparative problem where the student must identify why a specific machine learning model succeeds or fails. While the title of this post suggests it is a guide for lecturers, if you are a university student revising for your machine learning module, you have just found the ultimate revision strategy. By understanding exactly how and why we design these questions, you will know precisely what the mark scheme demands of you in the exam hall.

Examiners do not want to see rote memorisation. We want proof that if we hand you a messy dataset and a failing neural network in a professional setting, you know exactly which diagnostic steps to take. We construct questions to reveal your problem-solving process. By looking at the mechanics behind a well-written exam paper, you will learn how to deconstruct the scenarios we put in front of you and secure the highest possible marks.

1. Testing diagnostic skills over simple recall

Writing a question that asks “What is a Random Forest?” is poor exam design because it only tests basic memory. Any student can memorise a textbook definition of ensemble learning and write it down. A much stronger question presents a specific set of symptoms and asks the student to diagnose the underlying machine learning disease. We want to see if you can connect the theoretical concepts you have learned to a practical failure happening right in front of you.

Consider a typical exam scenario. We might tell you that a decision tree has been trained on a housing price dataset. We provide the performance metrics, showing that the training error is exactly zero, but the validation error is exceptionally high. The question will ask you to explain what has happened and how to fix it. This tests your understanding of the mechanics of the algorithm rather than just its name.

In this worked example, the examiner is testing your knowledge of overfitting. A top-tier student will explain that a training error of zero indicates the decision tree has grown too deep, creating individual leaf nodes for every single training instance. The model has memorised the noise in the training data and completely failed to generalise.

To get the full marks for the solution, you must suggest concrete mechanical fixes. You would state that the model requires pruning to limit its maximum depth, or that you need to set a minimum number of samples required to split an internal node. We design this type of question to see the direct link in your mind between the symptom of high variance and the mechanical cause within the algorithm itself.

2. Forcing a choice between competing algorithms

In professional practice, machine learning engineers rarely work with a single algorithm in isolation. They constantly evaluate trade-offs between different approaches based on the constraints of the data. Therefore, a good exam question forces you to choose between two competing algorithms and defend your choice using technical evidence. We are not looking for a single right answer. We are looking for the quality of your justification.

We might construct a scenario where you are given a medical text classification dataset containing 10,000 distinct features (words) but only 100 patient samples. We will ask you to choose between a Support Vector Machine with a linear kernel and a highly complex deep neural network, and to justify your decision.

If you select the linear Support Vector Machine, your justification must reference the dimensionality of the data. You should explain that in a high-dimensional space with very few samples, the data points are almost certainly linearly separable. Introducing a complex neural network will immediately lead to severe overfitting because the network has far more parameters than there are data points to constrain them.

Alternatively, if a student lists the general pros and cons of both algorithms without applying them directly to the specific 10,000-feature constraint provided in the prompt, they will fail to gain the application marks. We write these questions specifically to punish generic answers and reward students who read the parameters of the scenario carefully. Your technical vocabulary must directly address the specific dataset described.

3. Evaluating the mathematics in a practical context

Mathematics is the absolute foundation of artificial intelligence. However, asking a student to reproduce a raw mathematical derivation without any context is generally a weak way to test understanding. Instead, we design questions that ask you to calculate a single algorithmic step. This proves that you understand how the mathematics translates into the actual learning process of the model.

A classic example is asking you to perform one single step of gradient descent for a simple linear regression model. We will give you the starting parameters. Let us say the initial weight is 0.8, the learning rate is 0.1, the input value is 2, and the target value is 1. We then ask you to calculate the new weight after one iteration.

To solve this worked example, you must show the sequential mathematical steps. First, calculate the current output by multiplying the weight by the input, which gives 1.6. Next, find the error by subtracting the target from the output, giving 0.6. You then calculate the gradient by multiplying the error by the input, resulting in 1.2. Finally, you multiply the gradient by the learning rate to get the update value of 0.12, and subtract this from your original weight. Your final new weight is 0.68.

The examiner is watching closely to see if you add or subtract the gradient, and whether you apply the learning rate in the correct sequence. A minor arithmetic mistake might only lose you a single mark, provided your written steps are clear. However, a conceptual error, such as updating the weight in the wrong direction and moving away from the minimum loss, shows a fundamental misunderstanding of the calculus and will cost you heavily.

4. Assessing awareness of ethical and practical constraints

Modern machine learning exams must reflect the realities of deploying algorithms into live environments. We cannot just test the mathematics. We must test how these models behave when fed real human data. A strong exam question will introduce a scenario fraught with data leakage, computational limits, or demographic bias, and ask you to identify the primary risk.

We might present a scenario involving a predictive policing model or a CV-screening tool trained entirely on historical hiring data. The question will ask you to identify the risks of deploying this model and propose technical mitigations. We are testing whether you understand that algorithms are not neutral and that they mathematically amplify the assumptions present in their training data.

If the training data contains severe class imbalances, the model will naturally learn to associate minority groups with negative outcomes or simply ignore the minority class entirely to optimise for overall accuracy. A strong student will explicitly state that standard accuracy is a highly misleading metric in this scenario. You should explain that the model requires evaluation using precision, recall, or the F1-score for the minority class.

The mark scheme for this type of question looks for specific, actionable technical vocabulary. You will not score highly by simply writing that the model is unfair or biased. You must state that the model suffers from sampling bias which skews the decision boundary, and then suggest specific interventions like stratified sampling, applying synthetic minority over-sampling techniques, or implementing class weighting in the loss function to correct the imbalance.

How to answer this in an exam

When you sit down in the exam hall and face these carefully constructed scenarios, your first job is to identify the hidden concept being tested. Before you start writing, ask yourself what the examiner is actually looking for. Is this a question disguised as a business problem but actually testing the bias-variance tradeoff? Is it a question about image processing that is secretly testing your knowledge of vanishing gradients in deep networks?

University mark schemes heavily reward a specific structure. You should always start your answer with a direct, confident statement identifying the core problem. Follow this immediately with your technical explanation, ensuring you use the correct mathematical or algorithmic terminology. End your answer with a proposed solution or a clear justification of your choices.

The Full Marks Press Machine Learning guide covers this exact structure with worked exam questions, showing you side-by-side what a passing forty percent answer looks like compared to an excellent seventy percent answer. Seeing the contrast between a vague, descriptive answer and a precise, analytical one is the fastest way to improve your own writing style.

Always ensure you link your theoretical knowledge back to the prompt provided. If the exam question mentions a dataset of hospital patients and medical diagnoses, your written answer must explicitly mention patients and diagnoses. Do not write a generic answer about inputs and outputs. Grounding your technical knowledge in the specific scenario proves to the examiner that you can apply your skills to real problems.

Frequently asked questions

Q: How much mathematical derivation do I need to memorise for an AI exam? A: Focus heavily on understanding the mechanics of the formulas rather than blindly memorising them. You need to know how changing a specific variable, such as increasing the learning rate or altering the penalty term, mathematically alters the final outcome of the algorithm.

Q: Will I lose all the marks if my final calculation is wrong but my method is right? A: Most university mark schemes award partial credit for demonstrating the correct method. Clearly writing out your steps sequentially ensures you pick up these method marks, even if your basic arithmetic fails under the pressure of the exam clock.

Q: How do I know how much detail to include in a written answer? A: Always look at the mark allocation next to the question. A two-mark question requires a brief definition or a single justified point. A ten-mark question requires a heavily structured argument containing technical definitions, specific applications to the scenario, and comparisons of alternatives.

Q: Should I write pseudocode if the exam question does not explicitly ask for it? A: You should only write pseudocode if it is genuinely the fastest way to convey your understanding of an algorithm’s internal logic. Otherwise, writing a clear set of bullet points describing the algorithmic steps is usually safer, less prone to syntax errors, and much quicker to write.

You can find a free digital copy of our practitioner-written revision guides to help you prepare for your next paper at fullmarkspress.com/free.