Case studies in AI ethics: examples to use in your next exam
Excellent exam answers in AI ethics rely on specific, real-world case studies that demonstrate algorithmic bias, privacy violations, or accountability failures. The most reliable examples to use are the COMPAS recidivism algorithm for racial bias, the Amazon automated recruitment tool for historical data bias, and the Optum healthcare algorithm for proxy variable failures.
Memorising abstract ethical principles is only half the battle when preparing for your assessment. In an exam setting, a marker wants to see you apply these principles to concrete scenarios. You need to explain the technical failure that led to the ethical breach and suggest how machine learning engineers could have prevented it. The case studies below give you the precise details you need to construct high-scoring arguments.
They illustrate the most common pitfalls in machine learning development, ranging from selecting inappropriate proxy variables to scraping training data without informed consent. When you quote these cases accurately, you show examiners you understand how mathematical decisions translate into human consequences.
The COMPAS algorithm and competing definitions of fairness
The Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) system is the most famous case study for discussing algorithmic fairness. Developed by Northpointe, courts in the United States used this risk assessment tool to predict the likelihood of a criminal defendant reoffending. The ethical failure emerged when an independent analysis by ProPublica revealed the system was racially biased. It incorrectly flagged black defendants as high risk at nearly twice the rate of white defendants, while white defendants who went on to reoffend were more frequently mislabelled as low risk.
For an exam, the technical takeaway here is the conflict between different mathematical definitions of fairness. Northpointe argued the model was fair because it achieved predictive parity. A risk score of seven meant the exact same likelihood of reoffending regardless of the defendant’s race. However, ProPublica focused on the false positive rate. Because the base rates of arrest differed between populations due to systemic policing practices, satisfying predictive parity mathematically guaranteed a disparity in false positive rates.
A worked exam example might ask you to evaluate the fairness of a predictive policing system. You can use COMPAS to argue that a machine learning model cannot simultaneously satisfy all mathematical definitions of fairness when base rates differ across groups. You should point out that engineers must make a conscious, ethical choice about which metric matters most for the specific context. In criminal justice, minimising false positives for marginalised groups is often the more ethical choice to prevent unjust incarceration.
Amazon’s recruitment tool and the problem of historical data bias
In 2014, Amazon began building computer programmes to review job applicants’ resumes with the goal of mechanising the search for top engineering talent. The company trained its machine learning models on ten years of resumes submitted to Amazon. Because the technology industry is heavily male-dominated, the vast majority of these historical resumes came from men. The system quickly learned to penalise resumes that included the word “women’s” (such as “women’s chess club captain”) and downgraded graduates of two all-female colleges.
This case study is the perfect example to use when an exam question asks about training data bias and the limits of masking protected attributes. Amazon’s engineers attempted to fix the bias by programming the system to ignore explicit gender terms. This mitigation strategy failed entirely. The model simply found proxy variables for gender, such as the specific verbs applicants used. Male candidates frequently used words like “executed” and “captured”, which the model highly weighted based on the patterns in the historical data.
If you face a question about mitigating bias in natural language processing or recruitment algorithms, use this Amazon case to explain why simply dropping a protected class from a dataset does not solve the problem. You can argue that historical data reflects historical prejudices. When a model optimises for past success, it will inevitably reproduce the discriminatory patterns embedded in that data. Amazon eventually abandoned the project, which provides a strong concluding point for an exam essay. Sometimes the most ethical choice is to recognise a system cannot be fixed and to stop using it.
The Optum healthcare algorithm and proxy variable failures
A widely used commercial algorithm in the US healthcare system provides an excellent case study for the dangers of proxy variables. The algorithm, sold by Optum, was designed to identify patients with complex healthcare needs who would benefit from extra medical care. Researchers published a study in Science in 2019 showing the algorithm exhibited significant racial bias. At the same risk score, black patients were considerably sicker than white patients. If the bias were resolved, the percentage of black patients receiving extra help would have jumped from 18.7 percent to 46.5 percent.
The technical error in this case study is the selection of the target variable. The developers wanted to predict health needs, but “need” is a subjective, unquantifiable concept. Instead, they chose a proxy variable: future healthcare costs. The assumption was that sick people spend more money on healthcare. However, because of unequal access to care, lower income levels, and systemic biases in the medical system, black patients historically spent less on healthcare than white patients with the exact same level of illness.
When an exam question asks you to evaluate the design phase of a machine learning pipeline, bring up the Optum algorithm. It demonstrates that bias does not always come from explicitly malicious intentions or unbalanced training data. It often sneaks in during the problem formulation stage. You can earn high marks by explaining that engineers must critically evaluate the gap between the unmeasurable concept they want to predict and the measurable proxy they actually use. A good mitigation strategy to mention is conducting a disparity impact assessment before deploying a model to a live environment.
Clearview AI and the ethics of informed consent
For questions concerning privacy, data collection, and facial recognition, Clearview AI is an essential case study. The company built a facial recognition database by scraping billions of images from public websites, social media platforms, and employment sites. They then sold access to this database to law enforcement agencies and private companies. This allowed users to upload a photo of a person and find matching images along with links to where those images appeared online.
The core ethical violation here revolves around informed consent and the contextual integrity of data. People uploaded photos to platforms like Facebook or LinkedIn to connect with friends or find jobs, not to be entered into a global police lineup. Clearview AI argued that the data was publicly available, but privacy advocates and regulatory bodies strongly disagreed. The UK’s Information Commissioner’s Office fined the company for failing to have a lawful reason to collect the data and for not informing people what was happening to their biometric information.
In an exam, use Clearview AI to discuss the difference between public data and consensual data use. A strong answer will point out that just because a web scraper can collect information does not mean an organisation has the ethical or legal right to use it for machine learning training. You can also use this case to discuss the concept of biometric surveillance and the chilling effect such technologies have on civil liberties. Mentioning the General Data Protection Regulation (GDPR) principles, specifically purpose limitation and lawfulness, will directly hit the criteria on most UK university mark schemes.
How to answer this in an exam
Examiners are looking for more than just a retelling of a news story. A high-scoring exam answer in AI ethics follows a specific structure that demonstrates both technical understanding and ethical judgement. First, you must clearly identify the ethical issue in the prompt, whether it is data privacy, algorithmic bias, or a lack of explainability. State the issue concisely in your opening sentence.
Next, introduce your chosen case study as evidence. Do not waste time on irrelevant background details or corporate histories. Focus entirely on the technical mechanism that caused the failure. If you are discussing COMPAS, explain false positive rates. If you are discussing Optum, explain proxy variables. The mark scheme heavily rewards students who can connect a mathematical or design choice to a real-world harm. This proves you are thinking like an engineer who understands the consequences of their code.
Finally, you must evaluate mitigation strategies. A common trap is to state that the developers should just “make the data better” or “remove the bias”. You need to be highly specific. Discuss the limitations of dropping protected variables, the necessity of auditing tools, or the importance of having a human in the loop. The Full Marks Press AI Ethics and Responsible AI guide covers this with worked exam questions, showing you exactly how to structure your arguments to hit every requirement on the rubric. Conclude your answer by making a firm judgement on what the developers should have prioritised.
Frequently asked questions
How many case studies should I memorise for an AI ethics exam? You should prepare three or four highly detailed case studies. It is much better to deeply understand the technical failures and ethical implications of a few examples than to have a superficial knowledge of a dozen. Ensure your chosen cases cover different topics, such as racial bias, gender bias, and privacy.
Can I use hypothetical examples instead of real case studies? You can use hypothetical scenarios to explain a mathematical concept, but real case studies earn higher marks. Real examples prove to the examiner that you understand how these systems operate in production environments and the actual harms they cause to real populations.
What is the best way to explain proxy variables in an exam? Define a proxy variable as a measurable value used in place of an unmeasurable concept. Then, immediately give a concrete example. State that predicting healthcare costs instead of healthcare needs is a proxy variable failure, as it ignores systemic barriers that prevent certain groups from spending money on healthcare.
Do I need to know the exact statistics from these case studies? You do not need to memorise exact decimal points, but knowing the rough magnitude of the failure is highly beneficial. Stating that a system flagged one demographic at nearly twice the rate of another shows a strong grasp of the material and adds authority to your argument.
Pick up a free copy at fullmarkspress.com/free.