The legal system’s new evidentiary problem
Artificial intelligence is often discussed as a technical phenomenon, but its most consequential effects are now legal and institutional. Automated systems help rank defendants for pre-trial release, flag welfare fraud, support immigration triage, identify faces in public places, sift evidence and draft legal documents. Even where a human remains formally in charge, algorithmic outputs can shape perception, narrow discretion and create a veneer of objectivity that is difficult to rebut.
That matters because law is not merely concerned with accuracy. It is concerned with reasons, procedure and responsibility. A judge must explain a sentence. An agency must justify an administrative decision. A police officer’s powers are bounded by standards that can be reviewed. Once automated systems are inserted into these settings, long-standing legal safeguards begin to strain. Who bears the burden of explanation when a model is inscrutable? How does a claimant contest a score they cannot inspect? When an AI-enabled product causes harm, is the fault in design, data, deployment or oversight?
If an automated system can influence a person’s liberty, benefits or legal standing, its output cannot be treated as a mere technical detail.
The result is a new evidentiary problem for the rule of law. Modern legal systems must preserve due process while accommodating tools whose logic may be probabilistic, adaptive and hidden behind commercial secrecy. The emerging answer is not to ban all automation, but to make contestability, traceability and accountability central legal requirements.
From data protection to procedural justice
European law already contains a foundation for algorithmic due process. The General Data Protection Regulation provides individuals with protections in relation to automated decision-making, including, in certain circumstances, the right not to be subject to a decision based solely on automated processing that produces legal or similarly significant effects. It also provides rights to obtain meaningful information, express a point of view and contest decisions. These provisions are not a complete code for AI governance, but they establish an important principle: decisions that materially affect rights cannot be insulated from challenge merely because they are automated.
This principle has been reinforced in administrative and human-rights law. The Council of Europe has repeatedly stressed that algorithmic systems used in public administration and criminal justice must remain compatible with fair trial rights, equality before the law and effective remedy. Courts across Europe are gradually being asked to translate abstract rights into concrete obligations of disclosure, explanation and review.
The legal significance of this shift is easy to miss. For years, debates about AI ethics focused on broad values such as fairness and transparency. Law asks harder questions. Which party must produce evidence? At what threshold of impact does a right to explanation arise? What records must be kept? Under what circumstances may a trade-secret claim yield to a defence right or a due-process claim? These are not philosophical issues. They are questions of institutional design.
The right to contest automated decisions
The right to contest is becoming the practical hinge of algorithmic accountability. In theory, many public bodies insist that a human makes the final decision. In practice, however, a model’s recommendation may structure the decision so strongly that human review becomes nominal. A person denied a benefit, flagged as high risk or subjected to enhanced scrutiny may face severe consequences without any realistic opportunity to understand or rebut the automated contribution.
If an automated system can influence a person’s liberty, benefits or legal standing, its output cannot be treated as a mere technical detail.
European data-protection authorities have pressed for a meaningful, not cosmetic, form of review. A valid contestation process requires more than a generic statement that software was used. It requires enough information to identify the factors that mattered, the data sources relied upon and the route by which the individual can challenge errors, context or inference. This need not mean publishing source code in every case. But it does mean that explanation must be tailored to legal rights rather than to engineering convenience.
Courts are also beginning to test whether public authorities have delegated too much. A notable judgment from the District Court of The Hague in 2020 struck down the Dutch welfare-fraud detection system known as SyRI, finding that its operation lacked sufficient transparency and safeguards under the European Convention on Human Rights. The case did not establish a general prohibition on risk modelling in public administration. It did, however, show that opacity combined with intrusive state power is unlikely to survive serious judicial scrutiny.
Explainability as a legal standard, not a technical slogan
Explainability is often invoked loosely, as if any post-hoc description of a model were enough. In legal settings, that is insufficient. The question is not whether a system can produce a plausible narrative about itself. It is whether the explanation enables review, contradiction and remedy. A useful legal explanation must help answer at least four questions: what role the system played, what inputs were material, what error rates or limitations are known, and what human controls were applied.
In this sense, explainability is not a universal technical property but a context-sensitive legal standard. A recommendation engine for entertainment may tolerate vagueness. A tool used in sentencing or benefit allocation cannot. The stricter the effect on rights, the more demanding the explanation must be. This is why record-keeping, documentation and impact assessment are becoming so important. Without logs and audit trails, even a well-intentioned authority may be unable to reconstruct how a model affected an outcome.
In law, an explanation is adequate only if it enables challenge, review and, where necessary, remedy.
European legislation reflects this emerging approach. The AI Act imposes obligations for certain high-risk systems, including documentation, logging, human oversight and risk management. Although the Act is not itself a general procedural-rights charter, it strengthens the evidentiary infrastructure on which contestation and enforcement depend. It is easier to challenge a decision if deployers are required to keep records of how a system was configured and used.
Liability after harm
Rights to explanation and contestation address process; liability addresses consequences. When an AI-enabled system causes harm, injured parties often face acute asymmetries of information. The developer may control technical evidence, while the deployer may control operational records. The victim may know only that an automated tool was involved in a harmful outcome. Traditional liability rules do not disappear in such cases, but they can be difficult to use when causation is diffuse and evidence is inaccessible.
The European Union has tried to respond on two fronts. The revised Product Liability Directive updates product-liability law for the digital age by clarifying that software can fall within the concept of a product and by adapting disclosure and evidentiary rules. Alongside it, the proposed AI Liability Directive sought to ease certain burdens of proof in non-contractual civil claims involving AI, notably through disclosure measures and a rebuttable presumption of causality in defined circumstances. Although the legislative path of the latter has become uncertain, the legal problem it addresses remains plain: if claimants cannot access the information necessary to prove fault or causation, formal rights may be hollow.
This matters beyond consumer claims. Public authorities increasingly procure complex AI systems from external suppliers. If a policing tool contributes to an unlawful stop, or a case-management tool distorts access to justice, responsibility may be dispersed across the chain of design, integration and deployment. Liability law is therefore being asked to do more than compensate. It is being asked to force clearer allocation of duties ex ante: who validates data quality, who monitors drift, who trains users, and who must suspend use when risks become apparent.
Predictive policing and the revival of old biases
In law, an explanation is adequate only if it enables challenge, review and, where necessary, remedy.
Predictive policing has become a focal point because it dramatises the risks of statistical governance. Such systems may forecast places, times or persons associated with elevated risk, drawing on historical crime data and proxies that often encode the legacy of earlier policing patterns. If police patrol some neighbourhoods more heavily, they will record more offences there. A model trained on those records may then recommend yet more patrols in the same areas, creating a feedback loop that presents historical enforcement intensity as neutral prediction.
Research institutions and civil-liberties organisations have documented these dynamics repeatedly. The legal concern is not simply that a model may be inaccurate. It is that a system can repackage contested social choices as technical outputs, thereby making discrimination harder to see and contest. Under equality law and human-rights law, state action cannot be excused merely because a machine produced the ranking.
Risk scores in criminal justice raise similar concerns. Tools designed to estimate recidivism or failure to appear can influence bail, sentencing or supervision. Their defenders argue that they regularise discretion and may outperform unguided judgment. Their critics note that error rates can vary across groups, that the factors included may proxy socio-economic disadvantage, and that the resulting score can become a quasi-factual label in court. The core legal issue is not whether judges should use statistics at all, but whether the accused can inspect, challenge and contextualise the assumptions embedded in the score.
Due process in criminal proceedings
Criminal law sets the sharpest limits because the stakes include liberty and stigma. Fair-trial guarantees require defendants to know the case against them and to be able to test the evidence. When algorithmic tools are used to generate investigative leads, assess credibility or support sentencing, those guarantees become harder to secure. A proprietary score that materially shapes a judicial decision but cannot be interrogated sits uneasily with adversarial procedure.
American litigation around recidivism scoring and source-code disclosure has illustrated the tension, even if the doctrinal framework differs from Europe’s. European human-rights law is, if anything, more demanding in its insistence on effective challenge and equality of arms. The practical implication is not that every line of code must always be disclosed in open court. Rather, where an automated assessment plays a significant role, courts may need mechanisms for independent scrutiny, expert access, protective orders or exclusion where adequate testing is impossible.
Criminal procedure has long dealt with difficult forms of evidence, from forensic science to intelligence material. AI should be treated with at least equal caution. The lesson of past forensic controversies is that institutional confidence can outpace scientific reliability. If legal systems accept algorithmic outputs without rigorous validation, they risk repeating an old mistake in a more mathematical form.
The more authority legal institutions confer on a score, the stronger the case for giving those affected a genuine opportunity to interrogate it.
AI in the courtroom
The judiciary is also encountering AI as an internal tool. Courts and tribunals are experimenting with systems that assist in document review, legal research, transcription, translation, scheduling and the triage of routine claims. Properly governed, such tools may reduce delay and administrative burden. Yet they also raise delicate questions about judicial independence, transparency and the subtle migration of discretion from judge to software.
The most defensible uses are usually those furthest from adjudicative substance: summarisation under supervision, transcription checks, or workflow management. As tools move closer to recommending outcomes, legal risk rises sharply. A drafting assistant that suggests reasons or authorities may anchor a judge’s thinking, even if the judge remains formally responsible. If such systems are used, courts will need policies on validation, acceptable use, disclosure and record preservation. Litigants may reasonably ask whether an automated tool influenced the handling of their case.
Guidance from European judicial bodies points in this direction. The European Commission for the Efficiency of Justice has emphasised principles of transparency, non-discrimination, quality and user control in relation to AI in judicial systems. Those principles are useful, but they must be operationalised in mundane procedural rules: when use must be disclosed, how outputs are checked, what data can be entered, and how confidential material is protected.
The more authority legal institutions confer on a score, the stronger the case for giving those affected a genuine opportunity to interrogate it.
The procurement state and its accountability gap
Many of the most consequential legal risks do not arise from frontier research laboratories but from procurement. Public authorities buy, adapt and deploy scoring systems, biometric tools and analytics platforms through contracts that may obscure how decisions are made and who bears responsibility when things go wrong. Vendors may invoke confidentiality; agencies may claim limited technical understanding; individuals caught in the system are left facing a wall of institutional diffusion.
This is where public law and contract law intersect. Governments can require audit rights, documentation, performance testing, bias monitoring, incident reporting and data-governance standards at the point of purchase. They can insist on the practical ability to suspend or terminate use when rights risks emerge. They can also avoid contractual terms that undermine disclosure obligations in litigation or administrative review. Procurement, in short, is not merely an operational matter. It is one of the most powerful constitutional tools available to the modern state.
Yet procurement often lags behind legal principle. Authorities may assess cost and functionality more rigorously than due-process implications. As courts and regulators become more attentive, that imbalance is likely to prove expensive, both financially and institutionally.
What robust algorithmic accountability looks like
A credible framework for AI and justice does not require omniscience. It requires disciplined governance. First, institutions should classify systems by the severity of the rights they may affect. High-impact uses in policing, adjudication, welfare and immigration deserve stricter scrutiny than low-stakes administrative automation. Secondly, they should maintain documentation and logs sufficient to reconstruct a decision pathway. Thirdly, they should validate systems in their actual context of use, not merely rely on vendor claims or abstract benchmarks.
Fourthly, affected individuals need routes to contest decisions that are intelligible, timely and effective. Fifthly, independent oversight bodies and courts need access to the information necessary to investigate systemic harms. Finally, legal responsibility must be allocated across the lifecycle of a system: design, procurement, deployment, monitoring and withdrawal. One reason liability reform matters is that it pressures each actor to clarify its duties before failure occurs.
These measures may appear technocratic, but they protect classical legal values. Accountability is the architecture that allows liberty, equality and reasoned decision-making to survive institutional automation. Without it, public power becomes harder to see and therefore harder to restrain.
The next frontier is institutional, not merely technical
The debate over AI and justice is often framed as a race between innovation and regulation. That is misleading. The deeper challenge is institutional adaptation. Courts, legislatures, regulators and public authorities must decide how old principles apply when evidence is probabilistic, systems are supplied through private contracts and decision pathways are partly opaque. The law already has many of the relevant concepts: due process, equal protection, product safety, negligence, evidence, administrative review. What is changing is the context in which those concepts must operate.
Europe’s direction of travel is increasingly clear. Systems that affect legal rights will attract stronger duties of documentation, explanation, human oversight and redress. Automated authority will not be acceptable on the basis of performance claims alone. The burden will fall on institutions to show that a system can be governed, audited and contested in practice.
That is as it should be. Justice has never been only about getting to the right answer. It is about showing how power was exercised, on what basis and with what avenue for challenge. In the age of AI, the black box is not merely a technical inconvenience. It is a constitutional problem.
