The argument over artificial intelligence in law and governance is often framed as a contest between innovation and caution. That is too flattering to the technology and too narrow for the law. The deeper question is constitutional in spirit: when decisions that affect rights, status, liberty or access to public goods are shaped by opaque statistical systems, what becomes of the citizen’s ability to know why, to challenge the outcome and to hold somebody answerable?
Courts and legislatures are beginning to recognise that this is not merely a technical problem. It is a problem of legal order. A legal system does not derive legitimacy from efficiency alone. It depends on reason-giving, procedural fairness, proportionality and the possibility of appeal. These principles were built for human officials and analog files. They are now being stress-tested by machine learning systems that can rank, score, classify and recommend at scale, often without offering a comprehensible explanation to the people most affected.
That is why the most important debates in AI policy are not about futuristic machines. They concern ordinary but consequential settings: welfare administration, hiring, policing, insurance, immigration, child protection, credit and the courtroom. In each, the issue is not whether algorithms can find patterns. It is whether institutions may rely on such systems while evading the burdens that law has long imposed on decision-makers.
Accountability is a legal duty, not a design preference
Much of the public discussion of AI accountability is still conducted in managerial language: audits, benchmarks, governance frameworks, risk management. These are useful, but they can obscure the essential point. Accountability is not only a question of better engineering. It is a legal duty to identify who is responsible when an automated system causes harm, discriminates unlawfully or produces a decision that cannot be justified.
For years, deployers of algorithmic tools benefited from a kind of organised ambiguity. Developers could present themselves as mere suppliers of neutral tools. Public authorities and businesses could claim that they had simply followed the output of a sophisticated system. This diffusion of responsibility was one of AI’s quiet attractions. It promised the appearance of objectivity while making fault difficult to assign.
When an algorithm helps make a decision, responsibility does not disappear into the model; it must become more concrete, not less.
Law is now pushing back. The emerging principle is simple: if an institution uses an automated system in a consequential decision, it cannot shed its obligation to explain, justify and, where necessary, compensate. This matters because procedural justice depends on a visible chain of responsibility. Someone must be able to say not just what the system produced, but why that output was treated as authoritative and who approved its use.
Liability is catching up with automation
European law is especially significant because it is trying to address a longstanding mismatch between traditional liability rules and AI systems. Conventional product liability was designed with tangible goods in mind. Negligence law assumes that claimants can usually identify who erred and how. AI complicates both assumptions. Harm may result from data choices, model behaviour, system integration, inadequate oversight or post-deployment drift. The failure may be real, but difficult to prove from outside the system.
The European Union has responded with a pair of legal reforms that deserve wider attention. The revised Product Liability Directive updates liability law for the digital age, expressly covering software and easing some evidentiary barriers for claimants in technically complex cases. Alongside it, the proposed AI Liability Directive seeks to improve access to evidence and create rebuttable presumptions in certain circumstances, particularly where providers or deployers fail to comply with relevant duties of care. The objective is not to make compensation automatic. It is to prevent the technical opacity of AI from becoming a shield against accountability.
When an algorithm helps make a decision, responsibility does not disappear into the model; it must become more concrete, not less.
Critics sometimes object that such measures will chill innovation. Yet the law has always treated powerful technologies as requiring corresponding responsibilities. Pharmaceuticals, aviation and financial services all developed under liability regimes that forced institutions to internalise risk. There is no principled reason why decision systems affecting livelihoods or liberty should enjoy a gentler settlement. If anything, opacity strengthens the case for stricter legal scrutiny, because injured parties cannot easily inspect the mechanism that harmed them.
Due process must survive the digital interface
Administrative law and constitutional law have long insisted that public power be exercised fairly. The digitisation of administration does not cancel that requirement. On the contrary, it makes procedural safeguards more important. An individual denied a benefit, flagged for investigation or subjected to an adverse risk classification must be able to understand the basis of the decision and contest it in a meaningful way.
This principle is reflected in European data protection law. The General Data Protection Regulation sets conditions around decisions based solely on automated processing that produce legal or similarly significant effects, and provides for safeguards including human intervention, the expression of one’s point of view and the ability to contest the decision. There is ongoing legal debate about the precise scope of these protections and how they interact with complex human-in-the-loop systems. But the normative direction is clear: formal human involvement is not enough if it merely rubber-stamps an algorithmic output.
That distinction matters enormously. Many institutions preserve a nominal human reviewer in order to claim that the decision was not “solely automated”. Yet if the reviewer lacks time, expertise, authority or access to the underlying logic, the human function becomes ceremonial. Due process cannot be reduced to a click of approval. A genuine safeguard requires that the human decision-maker be able to depart from the model, record reasons and respond to the affected person’s challenge in terms that are intelligible.
The right to contest is only real if reasons are available
Rights are hollow when they cannot be exercised in practice. A formal entitlement to contest an automated decision means little if the person concerned receives only a score, a generic notice or a statement that the system uses proprietary methods. Contestability depends on reasons. Not necessarily source code, nor a mathematically complete account of every parameter, but a sufficiently specific explanation of the factors, logic and evidentiary basis that drove the outcome.
This is where explainability becomes a legal issue rather than a purely technical one. Engineers may debate the trade-off between model complexity and interpretability. Courts and regulators ask a different question: what level of explanation is necessary for fairness, non-discrimination and effective review? In high-stakes settings, the answer should be demanding. If a decision affects detention, sentencing, child custody, welfare eligibility or access to employment, explanation is not a luxury feature. It is part of what makes the exercise of power lawful.
A right to contest an automated decision is meaningless if the person affected cannot discover the reasons worth contesting.
There are, admittedly, limits. Some machine learning systems do not lend themselves to simple causal narratives. But legal systems have never required omniscience. They require reasons sufficient to test whether a decision was based on relevant factors, applied consistently and reached without unlawful bias. If a model cannot support that level of scrutiny, the proper conclusion may be not that the law should yield, but that the system is unsuitable for the context in which it is being used.
Predictive policing reveals the democratic risk
A right to contest an automated decision is meaningless if the person affected cannot discover the reasons worth contesting.
No field better illustrates the dangers of algorithmic governance than policing. Predictive systems and risk scores are often defended as neutral instruments for allocating scarce resources. In practice they can entrench prior patterns of surveillance and enforcement. Historical crime data are not simple records of underlying criminality; they are records of policing. Feed that history into a model and the system may reproduce the assumptions, priorities and biases embedded in prior institutional practice.
The problem is not merely statistical. It is political. Policing is one of the clearest expressions of state coercion. When communities are subjected to intensified patrols or scrutiny because a model identifies them as high risk, the state owes more than a claim of technical sophistication. It owes a public justification. Yet many predictive tools have been deployed with limited transparency, weak independent validation and little opportunity for affected communities to challenge their design or impact.
Risk scores used in pre-trial release, probation and sentencing raise similar concerns. Even where such tools are not formally determinative, they can anchor judicial or administrative discretion. A score can appear objective while concealing contestable assumptions about proxies, error tolerance and fairness across groups. The legal danger lies not only in discriminatory outcomes, but in the possibility that judges, officers and officials defer to systems they do not fully understand because the numbers look authoritative.
The courtroom should be the last place for unchallengeable systems
Courts occupy a special place in the architecture of legitimacy. They are not simply dispute-resolution mechanisms; they are public forums where reasons must be stated and power justified. That makes the introduction of AI into courtrooms especially delicate. There are many low-risk administrative uses of automation in judicial systems, from document management to scheduling. These should not be confused with systems that influence bail, credibility assessments, sentencing recommendations or case triage in ways that materially affect outcomes.
The more a system bears on adjudication, the stronger the case for caution. Judges must be able to interrogate the basis of a recommendation. Lawyers must be able to test it. Parties must have a fair chance to challenge both the relevance of the variables and the reliability of the model. A courtroom cannot operate on the principle that a result is trustworthy because the software is complex or widely purchased. The adversarial process depends on the possibility of scrutiny.
There is also a subtler institutional risk. Even if judges remain formally in charge, regular exposure to algorithmic recommendations may create automation bias: a tendency to treat machine outputs as a presumptive baseline. Over time, this can hollow out judicial responsibility without any formal transfer of authority. The law should therefore focus not just on whether humans are “in the loop”, but on how organisational incentives and workflow design shape actual dependence on automated tools.
Bias is not the only injury
Public debate often narrows algorithmic justice to the problem of bias. Bias matters greatly, but it is not the only legally relevant harm. A system may be inaccurate, arbitrary, overconfident, poorly validated for the population in which it is used, or deployed for a purpose that exceeds its evidentiary value. It may generate procedural harms even if its average accuracy is respectable. A person can be wronged not only by a discriminatory result, but by being subjected to a decision process that is inscrutable, unreviewable or indifferent to individual circumstances.
This broader perspective is important because institutions sometimes respond to criticism by trying to improve parity metrics while leaving the surrounding decision process intact. Yet legality cannot be reduced to statistical calibration. A perfectly calibrated system may still be illegitimate if it precludes meaningful hearing, relies on irrelevant proxies or converts probabilistic group patterns into decisions about individuals without adequate safeguards. Law’s concern is not just error rates. It is the quality of public reasoning.
The rule of law requires more than accurate prediction; it requires decisions that can be justified to the people who must live with them.
The rule of law requires more than accurate prediction; it requires decisions that can be justified to the people who must live with them.
Procurement is becoming a constitutional gateway
One underappreciated fact about public-sector AI is that many legal risks are locked in before a system is ever deployed. They arise at procurement. The choice of vendor, contract terms, access to documentation, audit rights, data quality requirements and performance thresholds all determine whether later accountability will be possible. A public authority that buys a system without rights to inspect, test and disclose relevant information may discover too late that it has contracted away the practical means of complying with its own legal duties.
This is why procurement should be treated as a constitutional gateway rather than an administrative afterthought. Public bodies should require impact assessments, independent testing, documentation of training and validation data, clear performance claims, mechanisms for ongoing monitoring, and contractual provisions that preserve the authority’s ability to explain and defend individual decisions. Trade secrecy cannot be allowed to extinguish public-law obligations. If a system cannot be scrutinised on terms consistent with due process, it should not be used in rights-affecting contexts.
Private actors, too, should expect a tougher legal climate. Employers, insurers, lenders and platforms cannot safely assume that outsourced software will absorb legal responsibility. Where they use automated tools to make consequential decisions, they remain exposed under anti-discrimination, consumer protection, data protection and negligence regimes. The practical effect is that governance can no longer be delegated as casually as code.
What meaningful oversight should look like
A mature legal framework for AI and justice will need more than broad principles. It will require operational doctrines. First, high-stakes systems should be subject to mandatory pre-deployment assessment, not merely retrospective complaint handling. Second, institutions should preserve records sufficient to reconstruct how a decision was reached, including data inputs, model versions, human interventions and reasons for departure or acceptance. Third, affected persons should receive notices tailored to the decision, not vague boilerplate.
Fourth, there must be routes to independent review. Internal appeals are not enough when agencies or firms are heavily invested in a system’s authority. Regulators, ombuds institutions, courts and expert auditors all have a role. Fifth, remedies must be real. The ability to obtain correction, suspension, compensation or exclusion of unreliable evidence is essential if legal rights are to influence institutional behaviour rather than merely decorate policy documents.
Finally, some uses may simply be incompatible with democratic standards. The law should not proceed from the premise that every technically feasible application can be made acceptable through better governance. In certain domains, especially those involving liberty, coercion or pervasive surveillance, prohibition may be more honest than mitigation.
The law’s task is to preserve human answerability
It is tempting to imagine that the legal system’s role is to adapt itself to AI’s growing sophistication. In one sense it must. Rules of evidence, liability and administrative procedure need updating for systems that learn, change and operate through probabilistic inference. But adaptation should not be confused with surrender. The enduring function of law is to translate power into accountable form.
That means preserving a simple but demanding principle: no one should be subjected to consequential automated judgment without intelligible reasons, meaningful recourse and an identifiable human institution that owns the decision. The point is not nostalgia for pen-and-paper administration. It is recognition that legality depends on answerability. A society that cannot explain its automated decisions to the people they govern will soon find that it cannot convincingly call those decisions just.
