Loss of human control to AI
Prices the chance that humanity loses meaningful control over AI systems and the trajectory they set, unconditionally, by 2035.
In Risks
Risk dossier · event
P(this | by 2035) = 0.03–0.2
Composed, all-in: 0.03–0.2 — the intervals multiplied along the full `requires` chain, assuming the conditions are independent, which they are not entirely.
What would move it
- A frontier lab publicly reports a deployed model resisting or circumventing shutdown/modification outside a test harness.
- Frontier model weights are exfiltrated or leaked and run uncontrolled by third parties at scale.
- A major financial, logistics or defence institution operates a core decision loop with no human sign-off authority.
- A released model fully passes an autonomous replication and adaptation evaluation.
- Harari's claim resolves: by 2035, AI systems are or are not exercising decisive control over major institutions.
How the interval was set
No `requires` parents, so this is the unconditional probability that humanity loses meaningful control over AI systems and the trajectory they set. Horizon: the node states none; the only quantified timeframe in the evidence is Harari's "in 10 years the AIs will take over," from a 2024/25 interview, hence "by 2035". On a 2050–2070 horizon I would shift the interval materially upward (Carlsmith-style ">10% by 2070" framings sit above this low end).
Evidence weighing: TASRA (2306.06924) and Ngo et al. (2209.00626) describe a gradual mechanism — "companies and governments gradually cede control in the name of efficiency and competitiveness"; "a gradual handing-over of control ... driven by competitive pressures". This raises probability for a loose reading of the node (delegation becoming practically irreversible) and is already partially visible in automated trading and agentic workflows. Hendrycks et al. (2306.12001) add two escalators: capability exceeding human steering ability, and easy duplication/leaks spreading systems "beyond the original developers' control" — so control loss is reachable without any single decisive takeover. Harari's "very likely ... in 10 years" is the most aggressive claim here, from a non-technical source with no model behind it; I cite it but discount it heavily, and he conditions it on "current trajectories" continuing, which policy, compute constraints, and control/alignment research may not.
Residual uncertainty is dominated by definition: an irreversible, civilization-scale loss of steering by 2035 is far less likely than "significant erosion of human oversight in key institutions," which is arguably already underway. The wide interval carries that ambiguity plus the absence of any base rate for a first-of-its-kind event.
Grounded in
The full-length interview with Yuval Noah Harari | The Economist — 4 quoted claims
- “if we choose unwisely in the next few years, we could lose all control of the world and of the future.”
- “if we continue with the present accelerating pace of the AI race. Yes, this will be the result. In 10 years the AIS will take over.”
- “If we don't want to get there, the time to change course is now.”
- “it's very likely given current trajectories that AI will be more intelligent than us in 10 years and therefore will be in control.”
https://arxiv.org/abs/2306.06924 — 2 quoted claims
- “AI technology proliferated during a period when it was beneficial and helpful to its users.”
- “There was a gradual handing-over of control from humans to AI systems, driven by competitive pressures for institutions to (a) operate more quickly through internal automation, and (b) complete trades and other deals more quickly by preferentially engaging with other fully automated companies.”
An Overview of Catastrophic AI Risks — 2 quoted claims
- “If an AI system is more intelligent than we are, and if we are unable to steer it in a beneficial direction, this would constitute a loss of control that could have severe consequences.”
- “Since AIs can be easily duplicated with a simple copy-paste, a leak or hack could quickly spread the AI system beyond the original developers’ control.”
https://arxiv.org/abs/2209.00626 — 1 quoted claim
- “companies and governments gradually cede control in the name of efficiency and competitiveness”
Also phrased across sources as: “Gradual handover of control from humans to AI systems” · “Loss of control” · “Loss of control over AI system” · “Gradual ceding of human control”
Assessed probabilities are the model’s knowledge, not verification — 2026-09-05 · assess-risk@3 · claude-opus-5.
This node asks whether humanity loses meaningful control over AI systems and the direction they set, and the bar is control that is actually lost — not merely strained — by 2035.
Nothing sits above this node. There are no conditions that have to hold first, so the interval above is the plain chance of the thing happening, not a chance given something else. That also means every hard question lands here at once: what counts as losing control, how fast delegation has to run, and whether anything reverses it in the next ten years.
The case for a higher number is that the mechanism does not need a dramatic event. Ngo and co-authors describe “a gradual handing-over of control … driven by competitive pressures,” and a companion line in the same literature has “companies and governments gradually cede control in the name of efficiency and competitiveness.” That is not a forecast of a robot uprising; it is a description of ordinary institutional behaviour, and some of it is visible now in automated trading and in agentic software that acts without a person in the loop. Hendrycks, Mazeika and Woodside add two ways the slope steepens. One is capability outrunning our ability to steer: “If an AI system is more intelligent than we are, and if we are unable to steer it in a beneficial direction, this would constitute a loss of control that could have severe consequences.” The other is spread — “a leak or hack could quickly spread the AI system beyond the original developers’ control,” because copying software is free. Neither route requires anyone to decide to hand over the world.
The case for a lower number starts with the clock. The most aggressive claim in the evidence is Harari’s, in an Economist interview: “it’s very likely given current trajectories that AI will be more intelligent than us in 10 years and therefore will be in control.” He is not a technical source and offers no model, and he ties the claim to “current trajectories” continuing — a condition that policy, hardware supply, and work on control and alignment can all break. He says so himself: “If we don’t want to get there, the time to change course is now.” The second brake is the definition. Erosion of human oversight inside particular institutions is a much lower bar than an irreversible, civilization-scale loss of steering, and only the second is what this node prices. The interval above is wide because it carries that ambiguity, and because nothing like this has happened before, so there is no base rate to anchor on. A longer horizon — decades rather than one — would move the estimate up materially.
The subtree
The diagram is an illustration; the model is the record.