AI and justice is now a constitutional question
Artificial intelligence has entered legal systems from several directions at once. Public authorities use automated tools to sort welfare claims, flag fraud, support immigration screening and prioritise police resources. Private firms deploy scoring systems that influence hiring, insurance, credit and access to services, with obvious knock-on effects for equality and economic rights. Courts themselves are experimenting with transcription, document review, research assistance and case-management tools. In each setting, the practical appeal is familiar: speed, scale and consistency.
Yet law is not chiefly concerned with speed. It is concerned with power: who exercises it, on what basis, with what safeguards, and with what remedy when things go wrong. That is why debates about AI and justice have moved rapidly beyond technical accuracy. A highly accurate system may still be unlawful if it discriminates indirectly, relies on opaque logic, prevents meaningful challenge or shifts responsibility into a contractual fog between developers, deployers and public bodies.
Automation may optimise administration, but justice requires reasons, accountability and an avenue of appeal.
The result is a broad legal realignment. Legislatures are crafting rules for safety and liability; data-protection authorities are testing limits on automated decisions; constitutional and administrative courts are asking whether algorithmic systems can satisfy due-process standards; and criminal-justice scholars are revisiting old assumptions about risk, evidence and discretion. The central issue is not whether machines can inform legal judgment. It is whether legal systems can preserve human responsibility when machine outputs become institutionally difficult to resist.
Algorithmic accountability means more than technical auditing
The phrase “algorithmic accountability” is often used loosely, as if it referred simply to better testing or more transparent coding. In legal terms, however, accountability is broader. It includes traceability of inputs and outputs, clarity over institutional responsibility, documentation of design choices, compliance with equality and data-protection law, and procedures for review and redress.
That broader understanding is visible in European regulation. The European Union’s AI Act classifies certain uses of AI as high-risk and imposes duties concerning risk management, data governance, technical documentation, logging, transparency, human oversight and post-market monitoring. Although the AI Act is not a complete justice framework, it signals a move away from voluntary ethics towards legally enforceable obligations. High-risk systems in law enforcement, migration, essential services and administration are treated not as ordinary software, but as technologies capable of affecting fundamental rights.
Accountability also depends on procurement and governance. A public authority cannot evade legal responsibility merely because a vendor built the model. If an agency relies on an automated score to allocate inspections, trigger investigations or support benefit suspensions, the legal burden does not disappear into the supply chain. Administrative law generally asks whether the authority acted lawfully, rationally and fairly. That inquiry now extends to model design, data quality, performance disparities and the real weight given to algorithmic outputs.
In practice, this means organisations need records, impact assessments and decision trails. A system that cannot explain what data it used, how often it fails, or who can override it is difficult to defend in court. Accountability, in short, is not a dashboard metric. It is the capacity to justify power under law.
Liability is shifting from abstract ethics to concrete legal risk
One reason AI governance is becoming more serious is that liability law is catching up. For years, many harms associated with automated systems sat awkwardly between product law, negligence, discrimination law and public-law review. Victims could often identify damage but struggled to prove causation, fault or access to evidence. European reform efforts are aimed partly at reducing that gap.
The revised EU Product Liability Directive updates strict-liability rules for the digital age. It expressly addresses software and certain digital manufacturing files, making it easier to frame claims where defective software causes damage. The logic is straightforward: if digital products can generate real-world harms, liability law should not be confined to physical defects in traditional goods. This matters for AI because system errors may stem from updates, data dependencies, cybersecurity weaknesses or emergent behaviour rather than from a cracked component or broken machine part.
Automation may optimise administration, but justice requires reasons, accountability and an avenue of appeal.
The proposed AI Liability Directive takes a different route. Rather than creating a wholly new liability regime, it aims to ease evidentiary hurdles in certain non-contractual civil claims involving AI. Its mechanisms include disclosure of relevant evidence in some circumstances and a rebuttable presumption of causality where a claimant can show fault and a plausible causal link to the output. The aim is not to guarantee compensation for every failure, but to address the asymmetry that often leaves claimants unable to prove what happened inside a complex model or socio-technical system.
These developments are significant because they recast AI harm as a justiciable matter rather than a regrettable by-product of innovation. They also spread incentives across the lifecycle: developers, deployers, importers and distributors all face stronger reasons to document testing, warnings, updates and oversight. In legal strategy, the question is moving from “should this be regulated?” to “who bears the cost when it fails?”
Automated decisions and the right to contest them
One of the clearest legal anchors in this field is the right not to be subject to certain decisions based solely on automated processing. In Europe, Article 22 of the General Data Protection Regulation provides a qualified protection where such decisions produce legal effects or similarly significant effects. Related provisions on transparency, access and meaningful information about the logic involved have fuelled wider arguments about algorithmic due process.
There are limits. Article 22 is narrower than many assume, and not every automated tool falls within it. A system that merely assists a human decision-maker may escape the category of a solely automated decision, even if the human review is perfunctory. That has made the factual question crucial: was there genuine human assessment, or was a person simply rubber-stamping the machine’s recommendation?
European case law is beginning to sharpen the point. In 2023, the Court of Justice of the European Union ruled in SCHUFA Holding that the automated establishment of a probability value concerning a person’s ability to meet payment obligations can fall within the GDPR rule where third parties draw heavily on that score to decide whether to enter into a contract. The decision matters beyond credit scoring. It recognises that formally indirect uses of an algorithm may, in practice, determine outcomes that significantly affect individuals.
For due process, the deeper principle is contestability. Affected persons need notice that automation was used, intelligible reasons for the result, and a realistic opportunity to challenge errors or unfair inferences. Contestability is not satisfied by a generic statement that “an algorithm assisted the process”. It requires enough information to make a challenge possible, whether by exposing inaccurate inputs, flawed assumptions, discriminatory proxies or institutional overreliance on scores.
A right to contest is hollow if the person affected cannot discover what was counted against them.
Explainability is becoming a legal expectation, if not an absolute right
Much has been written about a supposed universal “right to explanation”. The legal picture is more nuanced. Most jurisdictions do not grant a simple, freestanding right to a full technical explanation of any model. What the law more often requires is a bundle of adjacent rights and duties: reasons for decisions, transparency about the use of automation, access to personal data, non-discrimination, and effective remedies.
Even so, explainability is increasingly central in practice. Regulators and courts tend to ask whether a person can understand the basis of a decision sufficiently to challenge it and whether an organisation can explain its own system sufficiently to supervise it responsibly. That is not identical to opening every line of code. Often the legally relevant explanation is functional rather than source-level: what variables mattered, what thresholds were applied, what confidence limits exist, what error rates are known, and how human reviewers are instructed to treat the output.
The United Kingdom’s data-protection regulator and the Alan Turing Institute have published detailed guidance on explaining decisions made with AI, reflecting this practical orientation. Their work underscores a basic legal truth: opacity is not neutral. If a system is too complex to explain to those governing it, its use in rights-sensitive settings becomes difficult to justify.
There is, however, a tension worth noting. Some of the most powerful contemporary models are not inherently interpretable in a way that maps neatly on to legal demands for individualised reasons. Law therefore faces a choice. It can accept thinner explanations and rely more heavily on ex ante testing and institutional controls, or it can insist that systems used in high-stakes domains be interpretable enough to support individual challenge. Different sectors may choose differently, but the pressure is moving towards stricter explanation where liberty, livelihood or equal treatment is at stake.
Predictive policing exposes the limits of actuarial reason
A right to contest is hollow if the person affected cannot discover what was counted against them.
Predictive policing remains one of the most contentious intersections of AI and justice. Such systems use historical crime data or other proxies to forecast places, times or persons deemed at higher risk of offending or victimisation. Their defenders argue that police resources are finite and that data-driven allocation can improve efficiency. Critics counter that these systems risk laundering historical bias into a technical recommendation.
The criticism is not merely philosophical. If past policing was concentrated in particular neighbourhoods, the recorded data may reflect enforcement patterns as much as underlying crime. A model trained on those records can then direct yet more police presence to the same areas, generating a feedback loop in which surveillance reproduces itself. Similar concerns arise when “risk” is inferred from variables closely correlated with race, deprivation or prior contact with institutions.
Research and policy scrutiny have repeatedly highlighted these dangers. The European Union Agency for Fundamental Rights has warned about discrimination and accountability risks in predictive policing. In the United States, the National Institute of Standards and Technology and the National Academies have likewise stressed the need for careful evaluation, governance and context-sensitive safeguards in biometric and forensic technologies used by law enforcement.
The legal challenge is that risk scores can shape policing before any adjudication of guilt. They influence who is stopped, searched, monitored or prioritised, often with weak visibility for the affected communities. Traditional criminal-procedure safeguards are poorly suited to this front-loaded exercise of power. By the time a person reaches court, the algorithm may already have structured the path that brought them there.
For that reason, some jurisdictions and public bodies have paused, restricted or abandoned predictive tools after audits or public opposition. The broader lesson is that fairness in criminal justice cannot be reduced to calibration statistics. A system may satisfy one metric of accuracy and still corrode legitimacy if it entrenches selective enforcement or deflects democratic scrutiny.
Risk assessment in sentencing and bail raises a different set of worries
Risk scores in courts, parole and pre-trial release are often grouped with predictive policing, but they pose distinct legal problems. Here the issue is not simply where police patrol, but how judges and officials assess the likelihood of reoffending, absconding or breaching conditions. These tools promise consistency and evidence-based decision-making. Yet they also invite a subtle shift in legal reasoning: from judging past conduct to managing predicted future behaviour.
That shift is normatively fraught. Criminal law traditionally links punishment to culpable acts proved according to recognised standards. Risk assessment introduces population-level inference into decisions about liberty, often using variables such as age, employment status, housing instability or criminal history. Some may be statistically informative while remaining socially laden and ethically uncomfortable. Others can act as proxies for structural disadvantage.
The debate was sharpened by litigation in the United States over the use of proprietary risk-assessment software in sentencing. The Wisconsin Supreme Court’s decision in State v. Loomis allowed the use of a risk score with warnings and limitations, but the controversy underscored a global concern: can a defendant challenge a tool’s reasoning if its operation is opaque, proprietary or difficult to interpret? That question resonates in any legal system that permits algorithmic inputs into liberty-affecting decisions.
In criminal justice, a statistically useful prediction is not the same thing as a legally and morally acceptable basis for coercion.
Courts and legislators are therefore being pushed to distinguish assistance from substitution. A judge may consider a structured risk tool as one input among many, but if the score becomes dispositive in practice, due-process concerns intensify. The law tends to be most sceptical where an opaque model influences detention, sentencing or parole without robust disclosure, validation and the possibility of effective rebuttal.
AI in the courtroom may improve access, but it also changes judicial power
The use of AI in courts is not confined to criminal risk. Judicial systems are exploring tools for transcription, translation, document triage, legal research, scheduling and online dispute resolution. In principle, such technologies could reduce delay and administrative burden in overstrained systems. That matters because delayed justice is often denied justice, particularly in family law, housing, small claims and immigration.
Still, efficiency in court administration is not a neutral good if it comes at the expense of procedural fairness. Litigants may be poorly placed to detect when a generated summary omits nuance, when a translation shifts meaning, or when a triage system deprioritises certain claims. Judges, for their part, may face automation bias: a tendency to give excessive weight to machine-generated outputs, especially under time pressure.
In criminal justice, a statistically useful prediction is not the same thing as a legally and morally acceptable basis for coercion.
There is also an institutional concern. Judicial legitimacy rests partly on reason-giving, open justice and independence. If courts rely on systems they do not fully control or understand, these values may be weakened. The Council of Europe’s CEPEJ charter on the use of artificial intelligence in judicial systems and their environment insists on respect for fundamental rights, non-discrimination, quality, security and transparency. Its importance lies less in hard enforcement than in articulating standards by which courts can assess whether technology supports adjudication or subtly distorts it.
The prudent path is not to ban all AI from the courtroom. It is to separate low-risk administrative assistance from functions that bear directly on findings of fact, legal reasoning and credibility. The closer a tool comes to influencing adjudication itself, the stronger the case for disclosure, validation, human control and a presumption that the parties should know how the system was used.
Public law is becoming a crucial venue for algorithmic challenges
While liability and data protection matter, many of the most important contests over AI and justice are unfolding through public law. Claimants challenge automated or semi-automated systems on grounds familiar to administrative lawyers: illegality, procedural unfairness, irrationality, failure to take relevant considerations into account, and breach of fundamental rights. This route is often especially important where the state uses automation in welfare, immigration, taxation or policing.
Public-law scrutiny has a particular advantage. It does not require the claimant to fit every grievance into a narrow tort framework. Instead, it asks whether public power was exercised lawfully. That permits courts to scrutinise not only the final decision but also the surrounding institutional design: Was there a lawful basis for deploying the tool? Were equality impacts assessed? Was there a meaningful route to challenge? Did officials understand the system’s limitations? Were reasons adequate?
Such questions are likely to proliferate as governments automate frontline functions. The more the state relies on classification and prediction, the more legal systems will have to revisit old principles under new technical conditions. Fair hearing rights, open justice, reasoned decision-making and equality before the law were not drafted with machine learning in mind. Yet their underlying purpose is well suited to the task of disciplining automated power.
What a workable legal framework now looks like
A credible framework for AI and justice is beginning to come into view. It has at least six elements. First, clear allocation of responsibility across developers, deployers and public authorities, so that affected persons know whom to challenge. Secondly, sector-specific limits on high-risk use, especially in criminal justice and essential public services, where stakes are highest and error costs are asymmetric.
Thirdly, robust documentation and independent evaluation before deployment, including testing for accuracy, bias, robustness and foreseeable misuse. Fourthly, notice and contestability for individuals, with explanations pitched to legal relevance rather than technical theatre. Fifthly, human oversight that is real rather than nominal, backed by training, authority to depart from model outputs and institutional incentives not to rubber-stamp them. Sixthly, accessible remedies through courts, regulators and ombuds institutions, so that rights are not merely declaratory.
None of this guarantees perfect justice. Human decision-makers are themselves inconsistent, biased and opaque. But that is not an argument for algorithmic exceptionalism. It is an argument for parity of scrutiny. If public and private institutions wish to use AI where rights and liberties are on the line, they should be prepared to satisfy at least the standards long demanded of human authority: legality, intelligibility, equality and reviewability.
The next legal battle is over institutional memory
The most consequential question may be less about any single model than about what institutions forget when they automate. Legal systems contain forms of memory: the habit of giving reasons, the discipline of evidentiary testing, the visibility of discretion, the expectation that a person can face the case against them. AI can support those practices, but it can also erode them by making classification feel routine, neutral and too complex to contest.
That is why AI and justice should not be framed simply as a governance problem for technologists or compliance teams. It is a constitutional and civic problem concerning how authority is exercised over persons. The answer emerging across European and other democratic legal orders is cautious but clear. Automated systems may inform judgment; they may structure administration; they may even reduce some forms of arbitrariness. But where they affect rights, duties, liberty or equal treatment, they must remain subordinate to legal principles that are older and wiser than the technology itself.
The rule of law has always required more than correct outcomes. It requires processes that can be seen, understood and challenged. In the age of AI, that old requirement is becoming newly urgent.
