AI feasibility
Prices whether it becomes possible and affordable by 2070 to build AI systems that are highly capable, plan on their own, and model their own situation — the step every other state here rests on.
In Risks
Risk dossier · condition
P(this | by 2070) = 0.8–0.96
Composed, all-in: 0.8–0.96 — the intervals multiplied along the full `requires` chain, assuming the conditions are independent, which they are not entirely.
What would move it
- Frontier agents autonomously complete multi-week real-world professional projects at expert reliability without human checkpoints.
How the interval was set
Unconditional (no requires parents): probability that it becomes possible and financially feasible, by 2070, to build AI systems with advanced capability, agentic planning and strategic awareness — Carlsmith's premise (1), on which all five downstream nodes hang.
Evidence: the Carlsmith quotes here are all downstream conditionals ("conditional on (1)", "conditional on (1)–(4)"), so they mark this node's structural role rather than argue for it; his report assigns roughly 65% to premise (1) by 2070, and his later update revised the overall estimate to "greater than 10%" with the feasibility premise moved upward. That is the source's own number, and it is high and rising. Bengio supplies the affirmative mechanism: "a significant non-zero probability that the recipe for superhuman AI will build on what we have already discovered and that the missing pieces (which I believe are mostly system 2 abilities) will be uncovered within the next decade", plus the data-scale asymmetry ("learn from much larger datasets ... infeasible for humans in-lifetime"). His decade claim is deliberately hedged, but a ~45-year horizon absorbs decades of slack; feasibility needs only one success path.
No influences are recorded, so nothing pushes the interval sideways. What holds pHigh below 1: a genuine plateau in "system 2" reasoning and long-horizon agency; systems that are demonstrable but never financially feasible at scale; tail scenarios (severe civilizational disruption, durable global compute restriction) foreclosing the path. What holds pLow high: the long horizon, the permissiveness of "feasible", and observed agentic progress since 2021. Residual uncertainty is mostly about whether strategic awareness is a separate hard problem from raw capability — the evidence here does not settle that, which is why the band is 16 points wide rather than narrow.
Grounded in
Existential Risk from Power-Seeking AI — 6 quoted claims
- “It will be much harder to build aligned (and relevantly powerful/agentic) AI systems than to build misaligned (and relevantly powerful/agentic) AI systems that are still superficially attractive to deploy, conditional on (1) and (2).”
- “once some actors can create APS systems, then over time (and absent active efforts to the contrary) a larger and larger number of actors around the world will likely become able to do so as well”
- “Some such misaligned systems will seek power over humans in high-impact ways, conditional on (1)–(3).”
- “Such disempowerment will constitute an existential catastrophe, conditional on (1)–(5).”
- “There will be strong incentives to do so, conditional on (1).”
- “This problem will scale to the full disempowerment of humanity, conditional on (1)–(4).”
https://yoshuabengio.org/2023/06/24/faq-on-catastrophic-ai-risks/ — 3 quoted claims
- “computers can learn from much larger datasets (e.g. reading the whole internet) which is infeasible for humans in-lifetime”
- “there is a significant non-zero probability that the recipe for superhuman AI will build on what we have already discovered and that the missing pieces (which I believe are mostly system 2 abilities) will be uncovered within the next decade”
- “with system 2 machinery and objectives (which admittedly need to be scaled up) so that they can also reason better, be more coherent and imagine plans and counterfactuals.”
Also phrased across sources as: “Superhuman AI capability”
Assessed probabilities are the model’s knowledge, not verification — 2026-09-05 · assess-risk@3 · claude-opus-5.
This node prices one thing: whether, by 2070, it becomes both technically possible and cheap enough to actually do to build AI systems that are highly capable, that plan toward goals on their own, and that model their own situation well enough to act on it — the first step every other state in this subtree stands on.
The bar is low in one specific way, and that is what holds the bottom of the interval up. The question is not whether such systems get built, or deployed, or whether anyone wants them. It is only whether building them becomes possible and affordable. A single working path is enough, and there are forty-plus years for one to appear. Bengio names a concrete mechanism: there is, in his words, “a significant non-zero probability that the recipe for superhuman AI will build on what we have already discovered and that the missing pieces (which I believe are mostly system 2 abilities) will be uncovered within the next decade” — “system 2” meaning slow, deliberate reasoning rather than fast pattern-matching. He hedges that decade deliberately. The horizon here absorbs decades of slack, so the hedge costs less than it would over a shorter window. He also points at a standing structural advantage: computers “can learn from much larger datasets (e.g. reading the whole internet) which is infeasible for humans in-lifetime.”
The Carlsmith material in this dossier does not argue for this step so much as show what rests on it. Every quote of his is a conditional hanging off it — “There will be strong incentives to do so, conditional on (1)”; “This problem will scale to the full disempowerment of humanity, conditional on (1)–(4).” That is the shape of the thing: this node is the hinge. What Carlsmith does contribute directly is his own estimate. His report put roughly 65% on this premise by 2070, and his later update raised his overall figure to “greater than 10%” with the feasibility premise moved up. That is his number, from his own revision, and it moved in one direction.
What keeps the top of the interval short of certainty is not one big objection but three small ones. Progress on deliberate reasoning and long-horizon agency could genuinely stall rather than merely slow. Systems could be demonstrable in a lab and never affordable at the scale that “feasible” implies. And tail cases — severe civilizational disruption, a durable worldwide clamp on compute — would foreclose the path outright. Under all that sits the real open question: whether knowing enough about your own situation to act strategically is a separate hard problem from raw capability, or falls out of it. The evidence here does not settle that. That unsettled question is most of the width of the band above.
The subtree
The diagram is an illustration of the public subtree; the model is the record.