The argument over autonomy is asking the wrong question
The current discussion of AI agents is often framed as a contest between imminent transformation and overblown marketing. On one side are those who see software that can plan, call tools and execute tasks as the natural successor to today’s language models. On the other are sceptics who note, correctly, that systems still hallucinate, fail unpredictably and struggle with extended chains of reasoning. Both camps miss the more consequential issue. The question is not whether autonomous systems can sometimes complete useful tasks. It is whether they can do so reliably enough, audibly enough and economically enough to be embedded in serious organisations.
That distinction matters. Modern agentic systems already show glimpses of competence in bounded settings: triaging information, navigating interfaces, writing routine code, summarising cases, or orchestrating workflows across software tools. Yet each apparent success sits atop a stack of unresolved problems: fragile memory, poor calibration, unclear accountability, susceptibility to prompt injection, and a tendency to fail in ways that are difficult to predict in advance. What lies ahead, then, is unlikely to resemble a sudden march towards general autonomy. It will look more like the piecemeal industrialisation of agency.
“The decisive test for AI agents is not whether they can act, but whether their action can be bounded, inspected and interrupted.”
This is a less cinematic story than the one often told. But it is the one that will shape actual adoption. In aviation, finance, medicine and government, systems are not judged by their best demo. They are judged by their worst plausible failure, the cost of oversight and the clarity of liability. AI will face the same discipline.
From prediction to delegated action
The importance of agents lies in a technical and economic shift. Generative AI began by predicting the next token. Agents extend that predictive capacity into sequences of action: deciding what information to gather, which tools to call, which intermediate steps to take and when to stop. In practical terms, this means software no longer merely drafts an answer; it can be asked to perform a task within a digital environment.
That sounds incremental. It is not. The move from generating content to taking action changes the risk profile entirely. A mistaken paragraph can be ignored. A mistaken transaction, booking, recommendation, code deployment or medical instruction is a different matter. As the UK’s National Cyber Security Centre has warned in its guidance on prompt injection, tool-using language systems introduce fresh attack surfaces because untrusted external content can manipulate downstream behaviour. When systems are connected to email, databases, browsers and enterprise applications, every integration becomes both a capability and a vulnerability.
This is why the politics of autonomy will increasingly revolve around permissions, monitoring and fail-safe design. The architecture of trust will matter at least as much as the architecture of intelligence.
Benchmarks flatter, operations punish
Much public understanding of AI capability still derives from benchmark scores and staged evaluations. These are useful, but limited. A model can perform impressively on coding tasks, mathematical reasoning or question answering and still prove disappointing in deployment. Real work is messy. Goals are ambiguous, data are incomplete, users change their mind, systems time out and interfaces break. The challenge for agents is not simply problem-solving in the abstract; it is maintaining performance under operational friction.
The decisive test for AI agents is not whether they can act, but whether their action can be bounded, inspected and interrupted.
Research from institutions such as Stanford University’s Centre for Research on Foundation Models and Princeton University has repeatedly highlighted the gap between laboratory capability and real-world robustness. The same is true of broader work catalogued by the OECD and NIST on trustworthy AI. The core lesson is familiar from other technologies: once a system leaves the benchmark and enters an institution, edge cases become the main case.
For agents, this produces a harsh arithmetic. Every extra action in a chain creates another chance of compounding error. A system that is 95% reliable at each step may look strong in isolation, yet perform poorly over a long multi-step workflow. The longer the horizon of autonomy, the greater the need for verification, recovery mechanisms and human checkpoints.
Why supervision will remain central
Some enthusiasts treat human oversight as a temporary crutch, destined to disappear as models improve. That is improbable. In many sectors supervision is not merely a workaround for technical weakness; it is a structural requirement of governance. Boards, regulators and insurers will want to know who approved an action, what evidence was considered and how an exception was handled. Humans will remain in the loop not because machines can never improve, but because institutions require attribution.
This has an important implication for system design. The most valuable agents may not be those that fully replace human judgement, but those that sharpen it. A well-designed agent can assemble relevant documents, flag anomalies, draft alternatives, test scenarios and surface uncertainty. That may generate more value than a system that attempts end-to-end autonomy in settings where errors are costly and recourse is difficult.
“In high-stakes domains, oversight is not a temporary scaffold for autonomy; it is part of the product.”
There is a broader historical pattern here. Automation rarely eliminates management. It often redistributes it. As economists have long noted, new technologies can reduce some labour while increasing demand for monitoring, exception handling and coordination. Agentic AI is likely to do the same. The fantasy of frictionless delegation may be less durable than the reality of layered supervision.
The hidden bottleneck is memory and state
To work effectively over time, an agent needs more than fluent language. It needs memory: not only a transcript of prior exchanges, but an operational sense of goals, constraints, preferences and environmental state. This remains a major weakness. Context windows have expanded, and retrieval systems can fetch relevant information, but persistent and dependable memory is still brittle. Systems forget, overgeneralise, anchor on stale information or import irrelevant context.
These are not cosmetic defects. In practical deployments they can become serious liabilities. An agent that misremembers a user preference may be irritating; one that misremembers a compliance rule, patient detail or procurement threshold may be dangerous. The challenge is not simply storing more information. It is deciding what should be retained, how it should be updated, when it should be discarded and what degree of confidence should attach to it.
NIST’s AI Risk Management Framework is useful here because it directs attention away from abstract capability and towards lifecycle controls: mapping risks, measuring performance, managing impacts and governing change. For autonomous systems, memory is not merely a technical feature. It is a governance problem. Persistent state creates obligations around privacy, provenance, consent and auditability.
Security is not an add-on
In high-stakes domains, oversight is not a temporary scaffold for autonomy; it is part of the product.
One reason the agent debate has become overheated is that capability advances are easy to demonstrate, while security weaknesses are often slow, technical and inconvenient to explain. Yet security is where many ambitious visions will meet institutional resistance. Tool-using systems interact with untrusted environments. They can ingest malicious instructions hidden in documents or websites, leak sensitive information through indirect manipulation, or be induced to take unauthorised actions. The attack surface is broader than that of a standalone chatbot because the system is connected to operational tools.
The warning signs are already plain. Guidance from the UK’s National Cyber Security Centre and the US Cybersecurity and Infrastructure Security Agency underscores that prompt injection and data exfiltration are practical concerns, not science fiction. A system that autonomously browses, reads, decides and acts must be treated as a potentially compromised intermediary unless proven otherwise.
That implies a different engineering philosophy. Least-privilege access, sandboxing, permissioning, approval gates and detailed logging will matter as much as model quality. So too will organisational discipline: clear policies on what agents may access, what they may execute and what categories of action require explicit authorisation. In cybersecurity, prudence is not pessimism. It is infrastructure.
The economics favour narrow agency before general agency
There is also a simpler reason sweeping autonomy may arrive more slowly than expected: economics. Powerful models are expensive to run, difficult to tune and costly to supervise. If an agent requires repeated retries, lengthy verification or constant correction, its apparent labour-saving properties can evaporate. In many workflows, a modestly capable system that reliably handles a narrow slice of work will outperform a more ambitious agent that occasionally does everything and often does the wrong thing.
This is why the early gains are likely to accrue in constrained domains with clear interfaces, abundant feedback and low ambiguity: internal search, software testing, routine support, document handling, compliance triage and back-office operations. These are not trivial uses. They are economically meaningful precisely because they sit where standardisation is possible. But they do not require the grander narrative of digital employees endowed with broad discretionary judgement.
The OECD’s work on AI adoption has repeatedly stressed that organisational complements matter: data quality, process redesign, managerial capability and worker training. Agentic systems will be no exception. Their returns will depend less on abstract intelligence than on whether tasks are structured well enough for delegation in the first place. Many jobs are bundles of routine and judgement. Agents will first eat the routine.
Regulation will shape architecture, not just markets
As agents begin to act rather than merely answer, regulation will increasingly shape technical design. The European Union’s AI Act, though not focused solely on agents, points to a wider direction of travel: risk-based obligations, documentation requirements, transparency duties and heightened scrutiny for high-risk uses. Even where rules differ across jurisdictions, the underlying expectation is converging. Organisations deploying autonomous systems will need to explain what the system is for, how it is constrained and what happens when it fails.
This should not be read as a brake on innovation alone. Regulation often clarifies where innovation can proceed. Safety standards in aviation, pharmaceuticals and payments did not eliminate progress; they channelled it into forms institutions could accept. The same will be true here. The most successful agentic systems are likely to be those designed from the outset for traceability, reversibility and review.
“The race in AI autonomy will not be won by the system that can do the most, but by the one that institutions can justify using.”
This is particularly true in public services and other politically exposed sectors. A system that cannot produce an intelligible account of its reasoning and action trail will struggle to secure durable legitimacy, however efficient it appears in pilot settings.
The race in AI autonomy will not be won by the system that can do the most, but by the one that institutions can justify using.
What work will actually change
The labour effects of agentic AI are likely to be uneven and unspectacular at first, which is another reason public debate tends to misfire. Entire professions will not disappear overnight. Instead, tasks within professions will be redistributed. Junior knowledge work may change fastest because it often consists of information gathering, standard drafting, formatting, reconciliation and first-pass analysis. Agents can help here, albeit imperfectly.
But the result may not simply be substitution. It may also be compression. Fewer people may be needed for some forms of routine analytical work, while more demand emerges for workflow design, model oversight, quality assurance and domain-specific verification. In software, for instance, assistance with code generation may shift effort towards testing, architecture and security review. In legal or administrative contexts, the bottleneck may move from drafting to validation and accountability.
Economists at the IMF and researchers across academia have noted that AI’s labour effects will vary by task exposure and institutional readiness. The prudent conclusion is not that autonomy will either destroy or save work wholesale. It is that organisations able to decompose tasks carefully will capture gains, while those treating agents as magical general labour may absorb costs instead.
The strategic mistake is anthropomorphism
Perhaps the biggest conceptual error in the autonomy debate is anthropomorphism. Calling systems “agents” invites a misleading comparison with human colleagues: entities with stable goals, social judgement and contextual awareness. In reality, today’s systems are better understood as stochastic planners coupled to tools, retrieval mechanisms and guardrails. They may simulate coherence without possessing it in the durable way institutions require.
This matters because bad metaphors breed bad governance. If executives imagine autonomous software as a cheap employee, they may delegate too much too soon. If critics imagine it as a proto-person, they may argue about consciousness instead of control. The better analogy is to complex automation in other sectors: useful, sometimes transformative, but always in need of monitoring, maintenance and clear operating envelopes.
Language will eventually catch up with engineering. For now, policy and management should resist the seduction of human-like framing. Reliability, not personality, is the scarce resource.
A quieter and more consequential future
The most important effects of AI autonomy are therefore likely to emerge quietly. Not through theatrical demonstrations of machine independence, but through the gradual insertion of bounded agency into the plumbing of organisations. An agent that can reconcile invoices, collect evidence for an auditor, prepare procurement options, monitor system anomalies or draft case summaries may never look revolutionary to the public. Yet multiplied across thousands of processes, such capabilities could reshape administrative capacity, cost structures and managerial practice.
This quieter future is also the more plausible one. Institutions adopt technologies they can govern, not technologies they merely admire. The path to meaningful autonomy runs through constraint: scoped permissions, reliable memory, verification layers, strong security, intelligible logs and explicit human authority at points of consequence. What appears, at first glance, to slow progress may in fact be the condition for durable use.
That is why the debate needs a change of emphasis. Less attention should be paid to whether AI agents are on the cusp of becoming broadly human-like problem-solvers. More should be paid to the hard and unglamorous work of making software action legible, reviewable and reversible. Autonomy will matter when it becomes institutional, not theatrical. And institutional autonomy is built, as ever, through rules.
