Discussions of AI regulation can feel abstract. Principles such as fairness, transparency and human oversight are easy to endorse and hard to picture. The clearest way to understand why they matter is to look at what happens when they are missing.

The six cases below are among the best-known failures of automated systems in recent years. Each shows a different way in which AI can go wrong, and each helps explain a specific rule that European law now imposes on companies and public bodies.

1. The recruiting tool that learned to prefer men
In 2018, Reuters reported that Amazon had abandoned an experimental tool designed to rank job applicants. The system had been trained on CVs submitted to the company over the previous ten years, most of which came from men. It learned to penalise CVs containing the word "women's", as in "women's chess club captain", and downgraded graduates of two all-women's colleges. Engineers tried to correct the problem, but could not be sure that the system would not find other ways to discriminate, and the project was dropped.
What went wrong: the system reproduced the imbalances of the historical data it was trained on.
What the law says now: under the AI Act, systems used to recruit and select candidates are high-risk. From 2 December 2027, providers must ensure that training data are relevant and representative and examine them for possible biases. Employers using these systems must ensure human oversight, monitor their functioning and inform workers' representatives and the workers concerned before putting them into use. In Italy, the Transparency Decree (Legislative Decree 104/2022) already requires employers to inform workers when they use automated decision-making or monitoring systems.
The lesson: historical data describe the past, including its discrimination. Using them to automate future decisions requires testing for bias before and after deployment.

2. The benefits scandal that brought down a government
In the Netherlands, the tax administration used a risk-classification model to detect possible fraud in childcare benefits. Among the factors used was whether applicants had Dutch nationality. Tens of thousands of families were wrongly treated as fraudsters, ordered to repay large sums and, in many cases, pushed into serious financial hardship. The scandal led to the resignation of the Dutch government in January 2021, and the Dutch data protection authority fined the tax administration for processing nationality in a discriminatory and unlawful way.
What went wrong: a public body used an opaque risk model with discriminatory criteria, and people had no effective way to understand or challenge its results.
What the law says now: the AI Act classifies systems used by public authorities to assess eligibility for public benefits as high-risk. From 2 December 2027, public bodies deploying them must carry out a fundamental rights impact assessment before use. The GDPR already prohibits discriminatory processing and protects individuals against decisions based solely on automated processing.
The lesson: efficiency in detecting fraud cannot come at the cost of fairness and due process. The people affected must be able to understand and contest decisions.

3. The credit card accused of gender bias
In 2019, a well-known software entrepreneur complained publicly that his wife had received a much lower credit limit on the Apple Card than he had, despite their shared finances. Many similar complaints followed, and the New York State Department of Financial Services opened an investigation into the card's issuer, Goldman Sachs. In 2021 the regulator found no evidence of unlawful discrimination, but criticised the lack of transparency: customers could not find out the reasons for decisions that the bank was able to give the regulator.
What went wrong: even without proven discrimination, the inability to explain how credit decisions were made destroyed customers' trust.
What the law says now: AI systems used to evaluate creditworthiness are high-risk under the AI Act. From 2 December 2027, banks using them must carry out a fundamental rights impact assessment, and individuals affected by decisions based on high-risk systems have a right to an explanation of the role the system played. This complements the GDPR's safeguards on automated decisions.
The lesson: a decision that cannot be explained is difficult to defend, even when it is lawful.

4. The chatbot that made a promise
In 2022, a customer asked Air Canada's website chatbot about bereavement fares after the death of his grandmother. The chatbot told him he could apply for the reduced fare within 90 days of buying his ticket, which contradicted the airline's actual policy. When the airline refused the refund, the customer took the case to the Civil Resolution Tribunal of British Columbia. In February 2024, the tribunal rejected the airline's argument that the chatbot was a separate entity responsible for its own actions: the chatbot was part of the airline's website, and the airline was responsible for all the information on it. Air Canada was ordered to pay the difference in fare, plus interest and fees.
What went wrong: the company deployed a chatbot that gave customers wrong information, and then tried to disclaim responsibility for it.
What the law says now: the AI Act has required since 2 August 2026 that people be told when they are interacting with an AI system. In Europe, a company remains responsible for the information it gives its customers, whatever the channel, and misleading information about prices or conditions can amount to an unfair commercial practice under consumer law.
The lesson: a chatbot speaks for the company. Its answers must be tested, limited to what it can answer reliably and supported by an easy route to a human.

5. The algorithm that decided exam grades
In 2020, when exams in England were cancelled because of the pandemic, the exam regulator Ofqual used an algorithm to adjust the grades proposed by teachers for A-level students. Around 40% of grades were lowered, and students from historically lower-performing schools, often in disadvantaged areas, were more likely to be downgraded. After widespread protests, the government abandoned the algorithmic results within days and reverted to teachers' assessments.
What went wrong: a system designed to keep grades statistically consistent produced individual results that many perceived as unfair, penalising students for the past performance of their schools.
What the law says now: the AI Act classifies systems used to evaluate learning outcomes and to determine access to education as high-risk. From 2 December 2027 they will be subject to requirements on data, transparency and human oversight.
The lesson: a system can be statistically sound and still unacceptable if it treats individuals according to the group they belong to rather than their own merits.

6. The face database built from the internet
Clearview AI built a facial-recognition database by collecting billions of images from websites and social media without the knowledge of the people depicted, and offered it to law enforcement and other customers. Several European data protection authorities, including the Italian Garante in 2022 and the Dutch authority in 2024, imposed fines of tens of millions of euros and ordered the deletion of data relating to people in their countries.
What went wrong: personal and biometric data were collected on a massive scale without a legal basis, transparency or respect for individuals' rights.
What the law says now: since February 2025, the AI Act has prohibited AI systems that create or expand facial-recognition databases through the untargeted scraping of facial images from the internet or CCTV footage. The GDPR imposes strict conditions on the processing of biometric data.
The lesson: data that are publicly accessible are not free to use for any purpose.

What the cases have in common
The six cases involve different sectors, technologies and countries, but they share recurring patterns: data that encode past discrimination, systems that cannot be explained, humans who defer to the machine or are absent altogether, organisations that treat automated outputs as someone else's responsibility, and people who cannot contest decisions that affect them.
These are precisely the problems that the AI Act, the GDPR and national rules now address through requirements on data quality, transparency, human oversight, impact assessment and accountability. Some of these rules already apply: the prohibitions since February 2025 and the transparency obligations for chatbots since August 2026. After the Digital Omnibus on AI, most obligations for high-risk systems will apply from 2 December 2027. That leaves time to prepare, but not to postpone: systems being designed today are the ones that will be in use when the rules take effect.

For companies, the cases offer a simple test for any AI project: could this system produce a headline like one of these? If so, the time to find out is before deployment, not after. The practical question is not "could this happen to us?" It is "what in our processes would stop it from happening?"

‍

‍