Incentives to build powerful AI
Prices whether strong incentives to build powerful agentic AI will exist by 2070, given that such systems are technically and economically feasible.
In Risks
Risk dossier · condition
P(this | the condition below, by 2070) = 0.85–0.97
Conditional on
- AI feasibility — 0.8–0.96 by 2070
Composed, all-in: 0.68–0.93 — the intervals multiplied along the full `requires` chain, assuming the conditions are independent, which they are not entirely.
What would move it
- Frontier-lab or state annual AI capital expenditure continues rising year over year through the next decade.
- A major military or intelligence agency announces procurement of autonomous agentic AI decision-support systems.
- Enterprise revenue attributable to autonomous AI agents (rather than tool/assistant AI) exceeds that of non-agentic products.
- A binding international treaty restricts development of autonomous, strategically-aware AI systems and is ratified by leading developer states.
- Carlsmith or a comparable analyst publicly revises the premise-2 probability below 0.7 in an updated report.
How the interval was set
Conditioning: assume advanced, agentic, strategically-aware AI is technically and economically feasible by 2070 (the ai-feasibility parent). The question is then only whether strong incentives to build such systems exist.
Evidence is thin but convergent. Carlsmith states the claim as his premise 2 — "There will be strong incentives to do so, conditional on (1)" — and in that report assigns it roughly 80%, among his highest-confidence premises; several reviewers judged 80% conservatively low. Hendrycks, Mazeika & Woodside assert it flatly: "There will be strong incentives to build powerful AI agents." Neither source mounts a case for the negation.
The conditioning does most of the work. If systems with advanced capability and agentic planning are buildable and cheap enough to run, the value of automating labour and strategic decision-making, plus military and geopolitical competition, makes incentives near-structural. Observed behaviour through 2026 — frontier capex at national-programme scale, a race dynamic across labs and states toward agentic deployment — already confirms the incentive gradient before feasibility is settled.
Residual uncertainty keeping the upper bound below ~0.97: "strong incentives" is a soft predicate; a world could reach feasibility having internalised catastrophic-risk lessons via a hard international regime, a salient near-miss, or liability/insurance structures pricing agentic autonomy out. Alternatively the profitable path could route through non-agentic tool AI, weakening incentives for powerful agentic systems specifically. These are minority scenarios, and most would suppress building rather than the incentive itself. No influences were recorded to move the estimate either way.
Grounded in
Existential Risk from Power-Seeking AI — 5 quoted claims
- “It will be much harder to build aligned (and relevantly powerful/agentic) AI systems than to build misaligned (and relevantly powerful/agentic) AI systems that are still superficially attractive to deploy, conditional on (1) and (2).”
- “Some such misaligned systems will seek power over humans in high-impact ways, conditional on (1)–(3).”
- “Such disempowerment will constitute an existential catastrophe, conditional on (1)–(5).”
- “There will be strong incentives to do so, conditional on (1).”
- “This problem will scale to the full disempowerment of humanity, conditional on (1)–(4).”
An Overview of Catastrophic AI Risks — 1 quoted claim
- “There will be strong incentives to build powerful AI agents.”
Also phrased across sources as: “Strong incentives to build powerful AI agents”
Assessed probabilities are the model’s knowledge, not verification — 2026-09-05 · assess-risk@3 · claude-opus-5.
Given that advanced, agentic AI can be built and run at reasonable cost by 2070, this node asks whether anyone will have strong reasons to build it — and the bar is only that the incentives are strong, not that anyone acts on them.
Most of the hard questions were answered before this step. The parent condition already grants that systems with advanced capability and their own planning ability — agentic, meaning they pursue goals over time rather than answering one prompt at a time — are technically and economically within reach. Once that is granted, the pull toward building them is close to structural. Labour and strategic decision-making are the expensive parts of every firm and every state; a system that does them cheaply is worth an enormous amount to whoever holds it first. Military and geopolitical competition adds a second engine that does not switch off when one party decides to stop. What we can already see through 2026 points the same way: frontier training budgets at the scale of national programmes, and a race between labs and between states toward deploying agents rather than tools. The incentive gradient is visible before the feasibility question is settled.
The sources are thin in number but they agree, and neither argues the other side. Carlsmith states it as his second premise — “There will be strong incentives to do so, conditional on (1)” — and it is among the premises he holds most confidently, at roughly 80% in that report; several of his reviewers thought 80% was too low. Hendrycks, Mazeika and Woodside simply assert it: “There will be strong incentives to build powerful AI agents.” No source in the dossier mounts a case for the negation, and no separate influences were recorded that would move the estimate in either direction.
What keeps the top of the interval below certainty is mostly that “strong incentives” is a soft phrase, and a few worlds where it fails are conceivable. A world could arrive at feasibility having already absorbed the lesson from a bad near-miss, with a real international regime, or with liability and insurance rules that price autonomous agents out of the market. Or the money could turn out to sit in non-agentic tools — systems that answer and assist without running their own plans — which would weaken the case for powerful agents specifically. These are minority paths, and most of them suppress the building rather than the incentive: the reason to build stays, and something else stops it. That distinction is why the interval sits high and still leaves room.
The subtree
The diagram is an illustration; the model is the record.