The legal system’s algorithmic turn
Artificial intelligence has entered the justice system by degrees rather than through a single dramatic rupture. Automated tools now support credit scoring, welfare administration, migration control, policing, sentencing assistance, legal research and case management. In many jurisdictions, the most immediate legal issue is not the prospect of fully autonomous judging, but the subtler reality of administrative and judicial decisions increasingly shaped by prediction, classification and optimisation.
That shift matters because law is not merely a mechanism for generating outcomes. It is also a public process for giving reasons, allocating responsibility and allowing challenge. Traditional legal safeguards assume that a decision can be traced to an identifiable official or institution capable of explaining what happened and why. Algorithmic systems complicate that assumption. Their outputs may be probabilistic, their training data historically biased, and their internal logic difficult to articulate even for those who deploy them.
The result is a widening gap between computational capability and legal intelligibility. Justice systems are therefore being forced to answer a set of linked questions: who is accountable when an algorithm harms someone; what counts as a legally sufficient explanation; when must a person be allowed to contest an automated or partly automated decision; and how should evidential burdens be distributed when technical opacity impedes proof?
The legal problem with AI is not simply that machines may err, but that errors can become harder to detect, attribute and remedy.
From ethics to enforceable accountability
For much of the past decade, public debate about AI and justice was dominated by principles: fairness, transparency, non-discrimination and human oversight. Those ideas remain important, but they do not settle legal disputes on their own. Rights become meaningful only when institutions can investigate, standards can be applied and remedies can be obtained.
Algorithmic accountability is, at root, the translation of broad norms into enforceable duties. In administrative law, that may mean documenting procurement choices, validating a model before deployment, monitoring disparate impact and preserving an audit trail. In civil liability, it may mean identifying which actor in the chain of design, integration, deployment and supervision failed to meet a relevant standard of care. In constitutional and human-rights law, it means asking whether automation undermines equality before the law, procedural fairness or access to an effective remedy.
European institutions have become central to this shift from principle to procedure. The Council of Europe, the European Court of Human Rights and the institutions of the European Union have each, in different ways, emphasised that digital systems used by public authorities must remain compatible with legality, necessity, proportionality and the right to challenge state action. Similar debates are unfolding in common-law systems, often through data-protection litigation, judicial review and anti-discrimination claims.
Due process in an automated age
Few issues are more consequential than due process. A justice system worthy of the name must not simply reach decisions; it must do so through procedures that allow affected individuals to understand the case against them and to respond. Automated decision-making strains this model when the grounds of a decision are hidden behind proprietary code, technical complexity or institutional secrecy.
In Europe, one starting point is Article 22 of the General Data Protection Regulation, which provides individuals with protections regarding decisions based solely on automated processing that produce legal or similarly significant effects. The provision is narrower than many assume, and its precise scope remains contested. Even so, together with the GDPR’s information rights and general principles, it has supplied a legal vocabulary for challenging opaque decision-making in both public and private contexts.
Beyond data protection, due process concerns arise wherever automation influences entitlements, liberty or reputation. If a welfare recipient is flagged as high-risk for fraud, a visa applicant scored as suspicious, or a criminal defendant assessed as likely to reoffend, the affected person may face severe consequences without a clear account of how the risk score was generated. The law then confronts a familiar but sharpened problem: a person cannot meaningfully contest a case that is not intelligible.
Courts have increasingly recognised this tension. In Loomis in the United States, the Wisconsin Supreme Court allowed the use of a risk assessment tool at sentencing but acknowledged due-process concerns surrounding proprietary methodology and warned against overreliance. In the Netherlands, the district court in the SyRI case struck down legislation enabling a state risk-profiling system for fraud detection, finding that the regime violated the right to privacy under the European Convention on Human Rights because of insufficient transparency and safeguards. The broader lesson is clear: procedural rights cannot be reduced to a formal human sign-off if the human actor simply rubber-stamps an inscrutable score.
The legal problem with AI is not simply that machines may err, but that errors can become harder to detect, attribute and remedy.
Is explainability becoming a legal right?
The phrase “right to explanation” is often used too loosely. Most legal systems do not recognise a single, universal right requiring full disclosure of an algorithm’s inner workings in every case. Yet the absence of a neat doctrinal label should not obscure a more important reality: many legal regimes already require explanations of decisions, and those duties bite harder when automation is involved.
Administrative law commonly requires public authorities to give reasons. Data-protection law requires meaningful information in certain contexts. Fair-trial guarantees, anti-discrimination law and consumer law all create circumstances in which a person must be able to understand the basis of adverse treatment. In practice, what matters is not whether one can inspect every line of code, but whether the explanation is sufficient for scrutiny, challenge and redress.
That is a more demanding standard than superficial transparency. A legally useful explanation should identify the decision’s purpose, the factors that materially influenced the result, the role of human review, the data sources used, the possibility of error and the avenues for contest. For some systems, especially those built on complex machine-learning techniques, this may require layered explanation: a technical account for auditors and courts, and a plain-language account for affected individuals.
Explainability is becoming less a luxury of good governance than a precondition for meaningful contestability.
There is, however, a genuine trade-off. Requiring exhaustive technical disclosure in every case could reveal personal data, expose security-sensitive methods or place unrealistic burdens on public authorities. The legal challenge is therefore not maximal transparency, but adequate intelligibility relative to the severity of the decision and the risks of error.
Predictive policing and the risk of recursive bias
Predictive policing sits at the sharp edge of the AI-and-justice debate because it combines state coercive power with historically distorted data. Systems used to forecast where crimes may occur or who may be at heightened risk of offending are often trained on recorded crime, arrests or police contacts. Yet those data do not neutrally capture criminality; they capture patterns of reporting, patrol deployment and enforcement. Areas subject to heavier policing generate more police data, which can then justify yet more policing. The feedback loop can be self-reinforcing.
Legal concerns follow directly. If historical enforcement reflects racial, ethnic or socio-economic bias, predictive tools may reproduce that bias while cloaking it in mathematical neutrality. This raises questions under equality law, constitutional protections and human-rights standards. It also challenges the basic legitimacy of preventive interventions founded on group correlations rather than individualised suspicion.
Research and oversight bodies have repeatedly warned of these risks. The Council of Europe’s work on algorithmic systems and the criminal justice field has underlined dangers linked to discrimination, opacity and overreliance. Scholars at institutions such as the Ada Lovelace Institute and major law journals have similarly noted that predictive systems can convert social disadvantage into administrative suspicion.
The strongest legal objection to predictive policing is not merely that it may be inaccurate. It is that it can alter the threshold of state intervention in ways that are difficult to contest. A person may be subject to greater surveillance not because of proven conduct, but because of a probabilistic inference derived from patterns in data they never had the opportunity to examine or rebut.
Liability after the European reset
If rights are one side of the AI-and-justice equation, liability is the other. Harm caused by AI systems often sits awkwardly within legal frameworks designed around tangible products, identifiable defects and relatively linear chains of causation. Software updates, continuous learning, complex integration and distributed responsibility complicate the task of establishing who should bear loss when something goes wrong.
The European Union has responded through a broader attempt to modernise liability rules for the digital age. The revised Product Liability Directive, formally adopted in 2024, expands the concept of product to include software and eases some evidential barriers for claimants. It addresses situations in which digital products, including those with AI components, may be defective because they fail to provide the safety the public is entitled to expect. Importantly, claimants need not prove negligence; product liability remains a strict-liability regime in principle, though establishing defect and causation still matters.
Explainability is becoming less a luxury of good governance than a precondition for meaningful contestability.
Alongside this sits the more uncertain fate of the proposed AI Liability Directive, which was designed to help victims of AI-related harm, particularly through disclosure mechanisms and rebuttable presumptions concerning causation in certain fault-based claims. Whatever its eventual legislative trajectory, the proposal is significant because it confronts a core justice problem: where opacity makes proof unusually difficult, ordinary evidential rules may deny redress in practice even when wrongdoing is plausible.
These developments suggest a wider legal shift. The central question is moving from whether AI is exceptional to where existing doctrines fail under conditions of opacity and systemic complexity. In that sense, AI liability is not only about compensating harm. It is also about creating incentives for documentation, testing, monitoring and human oversight before harm occurs.
Who is responsible when many actors shape the outcome?
One reason AI liability is difficult is that harmful outcomes rarely stem from a single actor. A model may be developed by one entity, fine-tuned by another, embedded into a public authority’s workflow by a third and used by officials with varying levels of training and discretion. Data may be sourced elsewhere still. If the resulting decision is biased or unlawful, responsibility may be distributed across the chain.
Legal systems are accustomed to shared responsibility, but AI intensifies the problem. A public authority cannot escape scrutiny simply by invoking an external vendor, because public-law duties attach to the authority’s own exercise of power. Equally, a developer may argue that misuse or poor implementation by the deployer broke the causal chain. Courts and regulators will therefore need a more granular understanding of design choices, intended use, foreseeable misuse, model limitations and post-deployment monitoring.
This is why audit trails and impact assessments matter legally, not just operationally. They create the documentary basis for assigning responsibility. Without records of data provenance, testing results, override decisions and governance processes, claimants and judges are left to infer causation in the dark. That favours institutional defendants and weakens accountability.
In AI cases, liability often turns on documentation: if the system cannot be reconstructed, responsibility becomes easier to deny.
AI in the courtroom itself
Public concern often focuses on algorithms used by police or administrative agencies, but AI is also changing what happens inside courts. The most common applications remain relatively modest: transcription, document search, case triage, translation and scheduling. These may improve efficiency in overstretched systems. Yet as judicial institutions experiment more broadly, legal and constitutional concerns sharpen.
The courtroom is not an ordinary workplace. Its legitimacy depends on independence, equality of arms and reasoned adjudication. Any tool that influences how evidence is organised, which precedents are surfaced, how risks are scored or how cases are prioritised may shape outcomes indirectly even without replacing the judge. Automation bias is a particular danger: decision-makers may defer to system outputs, especially under time pressure, while still believing they exercised independent judgment.
There is also the problem of asymmetry. If one side in litigation understands the operation and limitations of a digital court tool while the other does not, procedural equality may suffer. If judges rely on AI-generated summaries or recommendations, parties may need disclosure about that use in order to challenge hidden errors. The older legal principle that justice must not only be done but be seen to be done acquires a technical dimension.
European guidance has tended to support cautious deployment. The European Ethical Charter on the use of artificial intelligence in judicial systems, issued by the Council of Europe’s CEPEJ, stresses respect for fundamental rights, non-discrimination, quality and security, transparency and user control. These are not mere aspirations. They point towards a practical judicial rule: digital tools may assist adjudication, but they must not erode the court’s duty to give reasons and remain answerable for the result.
Evidence, burden of proof and the opacity problem
Many disputes about AI in justice collapse into a simpler procedural question: who has to prove what, and how? If a claimant alleges that an automated system discriminated against them, produced a legally significant error or contributed to an unlawful arrest or sentence, the needed evidence may lie almost entirely with the deploying authority or the system provider. Source code, training data, validation studies and internal governance records are seldom publicly available.
In AI cases, liability often turns on documentation: if the system cannot be reconstructed, responsibility becomes easier to deny.
This asymmetry is not unique to AI, but it is amplified by technical complexity and claims of confidentiality. Conventional disclosure rules can therefore become decisive. So can presumptions that shift evidential burdens once a claimant establishes a plausible case. The proposed AI Liability Directive sought precisely to tackle this point by facilitating access to evidence and easing causation in defined circumstances. Even without that proposal, domestic courts may increasingly use existing procedural tools to compel disclosure where algorithmic opacity would otherwise make rights illusory.
For criminal proceedings, the stakes are higher still. If a defendant cannot examine the basis of a risk assessment, forensic tool or predictive model that materially affected the case, fair-trial rights may be compromised. The principle of adversarial testing is difficult to preserve when the operative reasoning is partly statistical and partly inaccessible. Courts will need to become more technically literate, but literacy alone will not solve the institutional imbalance. Procedural rights must be adapted so that opacity does not become a shield against scrutiny.
What a legally robust framework now requires
A workable settlement for AI and justice is emerging, though unevenly. It is not a simple ban-or-permit model. Rather, it rests on a handful of increasingly clear legal requirements.
- First, meaningful human responsibility must be preserved. Human oversight cannot be a decorative approval step; the responsible official or judge must have both authority and capacity to interrogate the system’s output.
- Secondly, documentation must be built in from the outset. Procurement records, impact assessments, validation studies, incident logs and override histories are essential for both governance and litigation.
- Thirdly, explanations must be calibrated to legal purpose. Affected individuals need understandable reasons and routes to challenge; expert bodies and courts may need deeper technical access.
- Fourthly, high-risk uses in policing, welfare and adjudication require especially strong safeguards because the costs of error are borne by rights, liberty and equal citizenship.
- Finally, liability and procedural law must account for opacity. If evidential burdens remain unrealistically high, formal rights will exist on paper but not in practice.
The wider implication is that law is not lagging technology quite as badly as is often claimed. Courts, regulators and legislatures are already shaping the conditions under which automated systems may be used in legally consequential settings. The real challenge is institutional capacity: judges, lawyers, oversight bodies and public authorities need the expertise and resources to convert abstract safeguards into routine practice.
Justice after automation
AI will not remove judgment from the legal system; it will redistribute it. Some discretion will be embedded in model design, some in data selection, some in threshold setting and some in the hands of officials who interpret outputs. That redistribution makes accountability more complex, but it does not make it optional.
The deepest risk is not that algorithms will openly replace law. It is that legal power will become more statistical, more preventive and less legible while retaining the appearance of procedural normality. A decision letter still arrives. A hearing still takes place. A judge or official still signs the outcome. Yet beneath those familiar forms, the grounds of decision may have shifted towards systems that are difficult to contest and easy to overtrust.
A serious jurisprudence of AI and justice must therefore defend more than privacy or innovation policy. It must defend the architecture of public reason: the proposition that coercive power should be explainable, attributable and open to challenge. In the end, algorithmic accountability is not a niche issue for technologists and regulators. It is a test of whether legal systems can remain recognisably lawful when authority is increasingly mediated by code.
