The legal problem is not automation alone
Artificial intelligence has entered the justice system in uneven but consequential ways: risk assessment in criminal procedure, predictive tools in policing, automated triage in public administration, transcription and document review in courts, and decision-support systems in welfare and immigration. These uses differ in purpose and maturity, but they raise a common legal problem. When a decision is influenced by a model that is opaque, probabilistic or trained on past institutional behaviour, traditional ideas of due process become harder to apply.
Law has long assumed that official decisions can be attributed to a person, justified by reasons, and challenged through a known procedure. Algorithmic systems strain each part of that chain. Attribution can blur when vendors, public bodies and operators each control a fragment of the system. Reasons can be difficult to extract when outcomes depend on complex statistical correlations rather than legible rules. Challenges can become more expensive and less effective when the affected person does not know that automation was used, cannot inspect the underlying logic, or lacks the evidence needed to prove error.
The key issue, then, is not whether machines will replace judges or police officers. It is whether legal systems can maintain accountability when human judgment is displaced, narrowed or quietly shaped by software.
The central legal question is not whether an algorithm is accurate on average, but whether a person can contest what it has done in their specific case.
Algorithmic accountability is becoming a legal standard
Algorithmic accountability is often discussed as a matter of ethics or good governance. In practice, it is becoming a legal standard assembled from existing doctrines: administrative law, anti-discrimination law, data protection, consumer protection, product safety and procedural fairness. These bodies of law do not use identical language, but they increasingly converge on a simple demand: if an automated system affects rights or material interests, someone must be answerable for its design, deployment and consequences.
In Europe, this trend is visible in the interaction between the General Data Protection Regulation, the EU Charter of Fundamental Rights and the Union's developing framework for AI governance and liability. Article 22 of the GDPR provides safeguards in relation to decisions based solely on automated processing that produce legal or similarly significant effects. Articles 13 to 15 require meaningful information in certain contexts about the logic involved, while broader fairness and transparency duties shape how public authorities and private actors must use personal data.
Outside data protection, courts and oversight bodies are asking older questions in newer contexts. Was there a lawful basis for relying on the tool? Were relevant factors considered? Was the decision reviewable? Did the system indirectly discriminate? Accountability, in other words, is not a single right. It is a bundle of legal requirements that together make automated power contestable.
Why liability law matters more than it first appears
Much public debate focuses on regulation before deployment: audits, conformity assessments and risk classification. Yet liability after harm matters just as much. Without a credible path to redress, rights can become largely theoretical. This is why the European Union's recent reforms to product liability deserve attention well beyond specialist circles.
The revised Product Liability Directive updates a regime designed for physical goods so that it can better address software and digital products. It recognises that software can be a product and that defects may stem from updates, cybersecurity vulnerabilities or failures in digital services connected to a product. This matters because AI-related harms may not always fit neatly into conventional negligence claims, especially when technical complexity and informational asymmetry make fault difficult to prove.
The central legal question is not whether an algorithm is accurate on average, but whether a person can contest what it has done in their specific case.
Alongside this, the proposed AI Liability Directive sought to ease the evidential burden in certain fault-based civil claims involving AI systems by introducing disclosure tools and a rebuttable presumption of causality in defined circumstances. Although the legislative path has been politically uncertain, the underlying legal problem remains clear: injured parties often cannot access the evidence needed to show how an AI system caused harm or whether a duty of care was breached.
Liability rules do not merely compensate after the fact. They also shape incentives before deployment. If public bodies, developers and deployers know they may have to explain how a system functioned, why safeguards failed, and who supervised it, they are more likely to build documentation, logging and oversight into the system from the start.
The right to contest is the backbone of due process
A justice system cannot be legitimate if people are bound by decisions they cannot meaningfully challenge. This principle is old; what is new is the way automated systems can frustrate it. In many administrative and quasi-judicial settings, people do not discover the role of automation until late in the process, if at all. Even when they do, a formal right of appeal may offer little protection if the basis of the decision is inscrutable.
European human-rights law and domestic public-law traditions provide a strong foundation here. The right to a fair hearing under Article 6 of the European Convention on Human Rights, the right to an effective remedy under Article 47 of the EU Charter, and principles of natural justice all point in the same direction: affected persons must be able to know the case against them and answer it.
In practical terms, the right to contest automated decisions requires more than a generic statement that AI was used. It implies notice, intelligible reasons, access to relevant evidence, a route to human review by someone with authority to change the outcome, and a procedure that is realistic in cost and time. A nominal review by an official who merely rubber-stamps the model's recommendation is not much of a safeguard.
A right of appeal is hollow if the person appealing cannot see the logic, data or assumptions that shaped the decision.
Explainability is not absolute, but it is increasingly juridical
There is a tendency to treat explainability as a technical aspiration rather than a legal entitlement. That is too narrow. Law does not always require disclosure of source code or a fully granular account of a model's internal workings. But it often does require an explanation sufficient for the person affected, the court or the regulator to understand the basis of a decision and assess its legality.
This is a contextual standard. A system used to sort internal paperwork may need only minimal explanation. A system used in bail, child protection, immigration enforcement or benefits eligibility demands far more. The greater the impact on liberty, livelihood or family life, the stronger the argument that explanation is a legal right rather than a policy preference.
Judicial decisions have reinforced this point. In the Dutch childcare benefits scandal, automated and risk-based fraud controls contributed to grave injustices later examined by courts and parliamentary inquiry. In the United Kingdom, the Court of Appeal in the Bridges case on live facial recognition scrutinised the legal framework, discretion and safeguards around police use of the technology. These cases differ factually, but each shows that legality turns not only on whether a tool works, but on whether its operation is bounded by intelligible rules and reviewable criteria.
Explainability, then, should be understood as relational. It is not solely about making a model interpretable to engineers. It is about making state action answerable to the people subject to it.
Predictive policing tests the limits of statistical governance
A right of appeal is hollow if the person appealing cannot see the logic, data or assumptions that shaped the decision.
Few domains illustrate the stakes more sharply than predictive policing. These tools typically use historical crime data, incident reports, location patterns or network analysis to forecast risk, prioritise patrols or identify individuals deemed likely to offend or be victimised. Supporters argue that they improve resource allocation. Critics reply that they can encode historical bias, intensify surveillance in over-policed communities and create feedback loops that make previous patterns look like objective truth.
The legal concern is not only discrimination, although that is central. Predictive policing also challenges core rule-of-law values. Individuals may be subjected to enhanced scrutiny not because of proven wrongdoing but because a statistical model places them in a risk category. Neighbourhoods may receive disproportionate police attention because the data reflect prior concentration of policing rather than underlying rates of harm. This can entrench unequal treatment while preserving a veneer of neutrality.
In 2023, the European Court of Human Rights ruled in Glukhin v Russia on the use of facial recognition to identify a protester, underscoring the implications of AI-enabled surveillance for privacy and democratic freedoms. While not a predictive-policing case in the narrow sense, it signals the wider judicial scrutiny likely to apply when police rely on data-driven systems that affect assembly, expression and personal liberty.
For legal systems, the lesson is clear: predictive tools used by police must be treated as exercises of public power, not mere operational aids. That means clear legal bases, necessity and proportionality tests, independent oversight, and records sufficient for later challenge.
AI in the courtroom demands restraint as well as efficiency
Court systems face undeniable pressures: backlogs, cost, uneven access to representation and growing volumes of digital evidence. AI tools may help with transcription, translation, document sorting, scheduling and legal research. Used carefully, such functions can improve administrative efficiency without dictating substantive outcomes. But the closer a system moves toward assessing credibility, predicting case outcomes or recommending sentences, the sharper the legal and constitutional concerns become.
Courts derive legitimacy from reason-giving, impartiality and the visible exercise of judgment. If adjudication is shaped by tools whose assumptions are hidden or whose outputs are treated as objective despite uncertainty, that legitimacy may erode. A defendant who hears that a risk score influenced a bail or sentencing decision is entitled to ask what variables mattered, whether proxies for race or class were involved, and how the score was validated for the relevant population.
The United States debate around risk assessment tools, sharpened by the Wisconsin case State v Loomis, illustrates the point. The court allowed use of a proprietary risk assessment tool with warnings about its limits, but the controversy exposed how difficult it is to reconcile secret or poorly explained analytics with criminal due process. Even where courts permit such tools, they rarely escape criticism if the affected person cannot probe methodology and error rates.
In courts, efficiency is a legitimate aim; opacity is not.
Human oversight works only if humans can genuinely intervene
Many legal frameworks rely on the idea of human oversight as a safeguard. This is sensible in principle, but weak in practice if the human reviewer is overburdened, insufficiently trained or institutionally inclined to trust the machine. Research in behavioural science and human factors shows that people can defer too readily to algorithmic outputs, especially when those outputs appear quantitative or scientifically grounded.
Meaningful oversight therefore requires institutional design, not just a person in the loop. Reviewers need access to the basis of the recommendation, authority to depart from it, time to do so, and incentives that do not punish disagreement. Logs should record when human operators overrode or followed the model and why. Procurement and deployment contracts should specify documentation, monitoring and audit obligations. Public bodies should be prepared to suspend or withdraw systems when evidence of drift, bias or operational misuse emerges.
This is especially important in high-volume settings such as benefits administration, border control and local policing, where frontline staff may rely heavily on automated triage. If human review becomes formulaic, the promise of accountability dissolves into paperwork.
In courts, efficiency is a legitimate aim; opacity is not.
Disclosure, documentation and evidence are the new procedural battleground
As AI-related disputes reach courts and tribunals, much of the contest will centre on evidence. Claimants, defendants and judges will ask practical questions: what system was used, what version, what training data, what performance metrics, what internal guidance, what override policy, what logs? These questions are not peripheral. They determine whether a claim can be proved at all.
This is why documentation has become so important in modern AI governance. Technical records, impact assessments, incident reports and audit trails are not simply compliance artefacts. They are the evidential infrastructure of legal accountability. Without them, public bodies may be unable to defend their decisions, and affected persons may be unable to challenge them.
The burden of proof also matters. Ordinary civil procedure often assumes that each side can obtain enough evidence to make its case. In AI disputes, that assumption frequently fails. The operator or developer holds most of the relevant information. This asymmetry is one reason lawmakers have explored disclosure duties and presumptions aimed at restoring procedural balance. Even where specific AI liability reforms stall, courts may increasingly adapt existing disclosure and adverse-inference doctrines to prevent opacity from becoming impunity.
Fairness cannot be reduced to technical metrics
Developers often present fairness through model metrics: false-positive rates, calibration, demographic parity, equalised odds. These can be useful diagnostics. But law asks broader questions. Was the underlying objective legitimate? Were less intrusive means available? Did the deployment context create discriminatory effects? Were affected communities consulted? Could the decision be appealed and corrected quickly?
A model can satisfy one mathematical fairness measure and still be legally troubling. For instance, equal predictive performance across groups does not answer whether the variable being predicted is itself shaped by biased enforcement. Nor does it resolve whether use of the model is proportionate in the first place. Legal fairness is institutional and normative, not merely statistical.
That distinction matters for judges and regulators. They should resist the temptation to outsource evaluative judgment to benchmark scores alone. Metrics can inform legal analysis, but they cannot replace it. The rule of law concerns power, reasons and remedies as much as performance.
The next phase will be built in procedure, not slogans
Much commentary on AI and justice swings between utopian efficiency and apocalyptic displacement. Neither is especially useful. The more plausible future is procedural: a gradual reshaping of how legality is evidenced, how decisions are explained, and how responsibility is allocated across public authorities, vendors, professionals and courts.
In that future, the decisive questions will be concrete. Was the system appropriate for this function? Was there a lawful basis for its use? What documentation exists? Who can inspect it? What rights does the affected person have to notice, explanation and review? What happens when the system fails? These are legal design questions before they are technical ones.
Justice systems do not need to reject AI wholesale to preserve legitimacy. But they do need to insist that any use of automated tools remains subordinate to principles that predate computation: fairness, equality before the law, reasoned decision-making and effective remedy. If those principles hold, AI may become an administrative aid. If they fail, automation will not modernise justice so much as obscure the exercise of power.
The debate, then, should not revolve around whether the black box is clever. It should revolve around whether the people subject to it can still see, question and correct what the state is doing in their name.
