Automated decision-making and human oversight
the lessons of the fine imposed on Uber
The Dutch Data Protection Authority has fined Uber 824.99 million euros for entirely automated decisions concerning drivers. This case serves as a reminder that an organisation cannot leave a decision with significant consequences for an individual to a system alone without reviewing the application of Article 22 of the GDPR.
(Uber has announced it is challenging the decision. The case is presented here by way of illustration of a legal principle, without prejudice to its outcome.)
A landmark penalty under the GDPR
The Dutch data protection authority, the Autoriteit Persoonsgegevens, has announced a fine of €824.99 million against Uber.
This is the largest fine ever imposed by this authority and the second-highest under the General Data Protection Regulation.
According to the authority, Uber is alleged to have made entirely automated decisions regarding its drivers, without the safeguards required by the GDPR.
Uber disputes this analysis and has announced its intention to appeal the decision.
What really happened at Uber?
Between 2018 and 2022, driver accounts were automatically deactivated in the event of suspected fraud or insufficient customer ratings.
In particular, the system monitored drivers’ behaviour and the ratings they were given. When it detected a situation deemed to be abnormal, the account in question could be temporarily suspended or, in certain circumstances, permanently deactivated.
According to the Dutch authority, these decisions were taken without any genuine human assessment and without the individuals concerned being provided with sufficient information.
Whilst the service was suspended, drivers were no longer able to access the platform or earn income through it.
The problem, therefore, did not lie solely in the use of an algorithm. It stemmed from the decisive role entrusted to the system in a decision with significant consequences for people’s professional lives and incomes.
A case arising from complaints by French lorry drivers
The investigation stems from complaints lodged by 171 French drivers with the Human Rights League.
The organisation forwarded its complaint to the French data protection authority. The investigation was subsequently carried out by the Dutch authority, as Uber’s European headquarters are located in the Netherlands.
This procedure illustrates how the one-stop-shop mechanism provided for under the GDPR works. Where data processing has a cross-border dimension, the authority of the Member State in which the company’s main establishment is located may act as the lead authority, in cooperation with the other relevant European authorities.
What does Article 22 of the GDPR say about automated decision-making?
Article 22 of the GDPR sets out the framework for decisions that produce legal effects or significantly affect a data subject, where they are taken in a fully automated manner.
The principle is as follows: this type of fully automated decision-making is, in principle, prohibited. It is only permitted in limited circumstances, notably where the decision is necessary for the performance of a contract, authorised by law, or based on the individual’s explicit consent.
And even in these permitted cases, the organisation must put safeguards in place. Depending on the situation, it must, in particular:
- clearly inform the individual of the existence of an automated decision and its general rationale; ;
- allow them to express their point of view; ;
- allow for genuine human intervention; ;
- to give them the opportunity to challenge the decision and have it reviewed.
In other words, the law does not seek to ban automation. It requires that, when a machine makes a decision that is important to a person, that person must not be left without recourse against the system.
The crux of the matter: what constitutes «genuine» human intervention?
This is the point that some organisations do not fully grasp.
Simply adding a human who approves the system’s decisions with a single click is not enough. Meaningful human intervention requires that the person responsible for monitoring can actually influence the outcome.
In practical terms, this means that it must:
- have the necessary information to understand why the system produced this result; ;
- to have the expertise to assess it; ;
- have the authority to challenge it and, if necessary, amend or cancel it.
A human who merely rubber-stamps an algorithm’s decisions, without having the power or means to challenge them, does not constitute genuine human intervention. It is merely a facade of supervision.
This distinction is crucial. Many systems that appear to «have a human in the loop» actually only have a human who approves decisions without being able to challenge them. Under the GDPR, this is not sufficient.
Why this goes far beyond digital platforms
It would be tempting to view the Uber case as one specific to tech giants. That would be a mistake.
Any organisation that automates decisions which have a real impact on people is potentially affected. Here are some common examples:
- the automatic screening of applications in a recruitment process; ;
- an automated score determining whether a loan is granted or refused; ;
- the automatic suspension or termination of a customer account; ;
- automatic prioritisation of cases (benefits, payments, claims); ;
- automated allocation of slots, resources, or pricing.
As soon as a system makes a decision on its own, with a significant impact on a person, the issues raised in Article 22 arise.
Is this decision permitted? Has the person been informed? Can a human actually review it? Can the person challenge it?
It is not the sector that matters, but the fact that a system makes decisions on its own, with real-world consequences.
How can you ensure your automated decisions comply with regulations?
For an organisation, the process is broken down into a few practical steps.
1. Identify your automated decisions
List all processes where a system produces a decision that has an impact on individuals, without human intervention or with purely formal intervention. This is often the most revealing step.
2. Qualify each decision
For each decision recorded, ask two questions. Does it have significant effects on the person? Does it fall under an authorised case (contract, law, explicit consent)? This classification determines your precise obligations.
3. Integrate security by design
Rather than adding human oversight after the fact, design the process so that a competent human can intervene from the start, and so that the person is informed and able to contest. These safeguards are far more effective, and much less costly, when they are thought out beforehand.
4. Document and inform
Keep a record of automated decisions, their legal basis and the safeguards put in place. Then clearly inform the data subjects of the existence of the automated processing and their rights.
Automation does not delegate responsibility
The Uber case isn't just a record penalty. It's a reminder that automating a decision does not delegate the responsibility that comes with it.
A system can suggest, sort, flag, and accelerate. But when a decision genuinely affects a person’s life, employment, income, or access to a service, the law requires that a human can genuinely intervene, and that the person is not left without recourse.
For your organisation, the issue is not about giving up on automation. It is about knowing precisely which decisions cannot be left to a system alone, and designing the safeguards accordingly. Human oversight built in from the start is infinitely cheaper than a nine-figure fine, or the lost trust of your users.
Start today by mapping your automated decisions. It is the starting point of any compliance effort, and often the most enlightening.
This article provides an editorial summary for informational purposes and does not constitute legal advice. The case mentioned illustrates the issues associated with automated decisions, but its lessons cannot be applied to a specific situation without analysing its context. For any specific legal questions, it is recommended to consult a qualified professional.
Foire aux questions (FAQ)
No. Article 22 regulates decisions based solely on automated processing of personal data when they produce legal effects or similarly significantly affect a person.
These decisions are prohibited in principle, but may be authorised in certain limited cases, notably when they are necessary for a contract, authorised by law or based on the explicit consent of the individual. They must then be accompanied by the safeguards provided for by the applicable framework.
This is an intervention that enables the person responsible for the audit to understand, evaluate and challenge the system's output.
She must have the information, skills and authority required to confirm, modify or cancel the decision. A simple validation click, without critical review or the power to act, is not enough. This is token supervision.
Potentially, yes. The automated rejection of a job application, credit scoring, the automatic suspension of an account or the prioritisation of cases may fall under Article 22 where the outcome produces a significant effect or plays a decisive role in the final decision.
It is not the sector of activity that matters, but the degree of automation, the data used and the consequences for people.
Start by mapping your organisation's automated decisions.
Identify the processes in which a system makes decisions autonomously, or in which human intervention remains purely formal, when an individual may be significantly affected. Then examine the legal basis for each decision and the safeguards put in place.
This mapping constitutes the starting point for compliance. It often reveals automated decisions that the organisation had not yet identified as such.
