When civilisational risk is discussed in public, attention gravitates towards spectacular failure. One scenario imagines a highly capable model pursuing a misaligned objective. Another pictures autonomous weapons compressing decision time until war outruns diplomacy. A third worries about biological design tools lowering the threshold for catastrophe. These dangers are real enough. Yet there is a less cinematic failure mode that helps explain why advanced societies remain vulnerable even after repeated warnings: organisational forgetting.
By this I mean the gradual loss of institutional memory about how high-consequence systems fail, how safeguards erode, and which precautions were adopted only after painful near misses. Forgetting is not simple ignorance. It is a recurrent process in which lessons are learned, partially codified, diluted through turnover, reinterpreted under budget pressure, and then discarded when a newer technology appears to belong to a different category. The result is familiar in aviation, finance, public health, nuclear governance and cyber security. It is now becoming central to artificial intelligence and autonomy.
The first line of civilisational defence is often archival rather than algorithmic.
Why this angle matters now
By mid-2026, the policy vocabulary around frontier systems has matured. Governments speak of risk tiers, evaluations, incident reporting, model governance and red-team testing. Defence establishments publish principles on responsible military uses of autonomy. Standards bodies define risk-management processes. International organisations warn that AI should be trustworthy, robust and human-centred. None of this is trivial. But a pattern is visible beneath the new language: institutions still struggle to preserve continuity between lessons learned in older high-risk domains and design choices being made in newer ones.
That gap matters because the most serious accidents are rarely caused by a total absence of knowledge. More often, they arise when knowledge exists somewhere but fails to travel: from engineers to executives, from operators to regulators, from one administration to the next, or from one sector into another. Organisational memory is therefore not a peripheral administrative question. It is a safety mechanism.
Risk accumulates faster than memory
Digital systems scale unusually quickly. Capabilities can be replicated across firms, agencies and borders at software speed. Institutional learning does not move that fast. It depends on committees, procurement rules, legal mandates, after-action reviews and training cycles. This creates a structural mismatch. A hazardous capability can diffuse globally in months; the norms and routines needed to govern it may take years to sediment. During that interval, societies rely on brittle assumptions: that skilled operators will compensate for design flaws, that voluntary commitments will survive competitive pressure, or that incidents in one domain will be noticed before they echo in another.
Existential-risk debates often describe a race between capability and control. Another way to frame the same problem is as a race between innovation and remembrance. If memory loses, control usually loses with it.
The long half-life of forgotten warnings
Every mature risk domain contains episodes that should have become permanent civic memory. Nuclear history records false alarms, misread sensors and dangerous ambiguity in command-and-control. Public-health history records the costs of delayed reporting, fragmented surveillance and weak cross-border co-ordination. Financial history records the tendency of models to convert uncertainty into spurious confidence. Aviation safety records the value of anonymous reporting, checklists and non-punitive investigation. Cyber security records that complex socio-technical systems are attacked at their interfaces and through their dependencies, not only at their core.
The first line of civilisational defence is often archival rather than algorithmic.
None of these lessons maps perfectly on to AI or autonomous weapons. But the point is not analogy for its own sake. It is that high-consequence technologies repeatedly fail through combinations of tight coupling, opacity, incentive distortion and overconfidence in monitoring. New sectors often behave as though these interactions were novel.
A society that cannot remember its near misses will eventually repeat them in a more automated form.
Alignment is also an institutional property
In technical discourse, alignment usually refers to whether an AI system reliably pursues intended objectives while remaining corrigible, interpretable or otherwise controllable. That work is essential. But there is a parallel institutional question: are the organisations deploying such systems aligned with their own stated safety goals when incentives turn adverse?
An institution may publish prudent principles and still drift into misalignment if promotion structures reward speed over verification, if procurement rewards low upfront cost over lifecycle assurance, or if legal accountability is so fragmented that everyone can plausibly claim the critical judgement belonged elsewhere. In that sense, alignment failure is not merely a property of models. It can be a property of bureaucracies.
This matters acutely in military and security settings, where the pressure to reduce decision latency is strong. If human oversight becomes nominal rather than substantive, systems may remain formally compliant while functionally autonomous in all the ways that matter. The central risk is not merely that machines exceed human control, but that institutions lose the capacity to notice control slipping.
What autonomy does to memory
Autonomy alters the texture of evidence. In older systems, operators often developed tacit understanding through repeated direct interaction with machinery. Failure signatures were imperfectly understood, but they were at least legible to experienced humans. In software-intensive systems, especially those incorporating machine learning, operational knowledge can become more distributed and more perishable. Data scientists understand one layer, mission planners another, contractors another, regulators another. No single actor necessarily sees the whole chain.
That fragmentation weakens institutional memory in two ways. First, it makes post-incident learning harder because causal explanations are contested across organisational boundaries. Secondly, it impairs pre-incident learning because warning signs that would once have been obvious to veteran operators are now hidden inside performance dashboards, proprietary tooling or changing model versions. Memory cannot persist if the object of memory is itself unstable.
The peril of clean-sheet thinking
Fast-moving technical sectors tend to admire first-principles reasoning. There are virtues in that style. It can strip away stale assumptions and reveal where inherited rules no longer fit. But in civilisational safety, clean-sheet thinking has a vice: it tempts institutions to treat every generation of technology as unprecedented enough to warrant procedural amnesia.
A society that cannot remember its near misses will eventually repeat them in a more automated form.
One sees this when incident reporting is framed as an optional burden rather than a core safety asset; when audit trails are incomplete because product cycles were prioritised; when testing is designed around benchmark performance rather than misuse, cascading failure or degraded modes; and when the burden of proof quietly shifts from deployers to those warning of harm. Such choices are often defended as pragmatism. In aggregate, they amount to memory loss.
The record from other sectors suggests that severe harms become governable not when institutions are most confident, but when they become disciplined about preserving dissent, anomaly reports and operational history. Checklists, black boxes and confidential reporting channels are all technologies of memory.
Weapons autonomy sharpens the problem
No area illustrates the stakes more starkly than military uses of AI and autonomy. Here, errors can propagate at machine speed under adversarial conditions, with secrecy limiting external scrutiny and wartime urgency weakening procedural restraint. Official declarations increasingly stress responsible use, testing, human judgement and traceability. These are important commitments. Yet the hardest question is whether they can survive contact with crisis.
History offers an uncomfortable lesson: in high-tempo security environments, safeguards that depend on individual discretion are vulnerable to compression. If communications degrade, if data are ambiguous, or if leaders fear being outpaced, there is strong pressure to delegate more authority to automated systems. That pressure may be temporary in intention and permanent in effect, because emergency measures often harden into precedent. Organisational memory then changes direction. Instead of preserving restraint, it preserves the memory that bypassing restraint once seemed necessary.
The central risk is not merely that machines exceed human control, but that institutions lose the capacity to notice control slipping.
Public health offers a parallel
The same logic appears outside warfare. Pandemic preparedness repeatedly suffers from cyclical attention. After each emergency, reports document weak stockpiles, fragmented authority, brittle supply chains and delayed information-sharing. Reforms follow, then attention fades, capacities atrophy and expertise disperses. The world does not forget in the abstract; it forgets operationally. The knowledge still exists in documents, but the personnel, budgets and routines that made it actionable have thinned out.
Civilisational risk from AI may follow a similar pattern if governance relies too heavily on white papers and too lightly on durable institutions. An evaluation regime that exists only for one administration, one budget cycle or one model generation is not memory. It is a pause.
What robust memory looks like
Institutional memory should not be confused with simple record-keeping. Robust memory has at least four features. It is cumulative, so that incident data and lessons learned are preserved in comparable form over time. It is transferable, so that knowledge survives staff turnover and contractor churn. It is contestable, so that internal dissent and external critique are archived rather than erased. And it is actionable, meaning that remembered lessons are tied to authority, thresholds and rehearsed responses.
The central risk is not merely that machines exceed human control, but that institutions lose the capacity to notice control slipping.
In practical terms, this points towards mundane but powerful instruments: standardised incident taxonomies; protected reporting channels; audit logs that are meaningful to investigators rather than merely voluminous; retention of red-team findings; procurement rules that demand post-deployment monitoring; and regular exercises that simulate ambiguity, deception and cascading failure rather than only ideal conditions. None is glamorous. All are central.
The politics of remembering
There is also a political economy to forgetting. Memory can be inconvenient. It slows deployments, complicates narratives of inevitability and creates discoverable evidence of ignored warnings. Organisations therefore have recurring incentives to remember selectively. They celebrate successful innovation while treating adverse events as isolated, proprietary or reputationally sensitive. This is one reason independent oversight matters in every hazardous domain: not because officials or firms are uniquely negligent, but because institutions under competitive pressure tend to discount low-probability, high-impact warnings until they become expensive enough to command attention.
For civilisational risk, the consequence is stark. If the archive of failure remains fragmented or inaccessible, each new entrant behaves as though it is operating near the frontier of possibility rather than inside a long history of recurring governance errors. The same categories return under fresh terminology: weak escalation paths, unclear accountability, unsafe defaults, overextended operators, and confidence unsupported by evidence.
Remembering without freezing progress
There is an understandable fear that stronger safety governance amounts to technological paralysis. In practice, the opposite is often true. Mature safety cultures do not prohibit development; they make development legible, reviewable and therefore more sustainable. Aviation did not become safer by rejecting flight. It became safer by institutionalising memory about failure. Medicine did not advance by abolishing adverse-event reporting. It advanced partly because harms could be surfaced and studied.
The same principle should apply to transformative digital systems. The narrow path is not between innovation and stagnation. It is between progress that compounds knowledge and progress that compounds amnesia. The former can absorb shocks and improve under scrutiny. The latter appears efficient until the day it encounters a failure it had already been warned about, somewhere, by someone, years earlier.
The real bottleneck
Many discussions of existential safety still assume that the decisive bottleneck will be scientific understanding: better interpretability, stronger evaluations, more reliable control. Those are indeed bottlenecks. But mid-2026 has made another constraint impossible to ignore. Even where technical tools exist, institutions often lack the memory architecture to use them consistently across time, jurisdictions and crises.
That is why organisational forgetting deserves to be treated as a civilisational risk factor in its own right. It converts manageable hazards into repeated surprises. It severs present decisions from past evidence. And it invites societies to believe that because a threat is new in form, it is new in kind.
If there is a sober lesson from the history of high-consequence systems, it is this: catastrophe is seldom born from ignorance alone. More often, it emerges when warning, procedure and restraint are allowed to decay faster than capability grows. In the age of advanced AI and autonomy, remembering may prove to be one of the highest forms of safety.
