# Catastrophic, not existential In spring 2023, two widely signed public declarations set the terms of the AI-risk debate: a March open letter calling for a pause on giant AI experiments, and a May statement from the Center for AI Safety that mitigating "the risk of extinction from AI" should be a global priority. Andrew Maynard declined to sign either — and published a reasoned alternative to each. His position, built on a career in risk science running from aerosols through nanotechnology to transformative AI and other transformative technologies, holds two commitments the 2023 debate treated as opposites: AI risk genuinely belongs among the world's priorities, and the extinction framing is the wrong vehicle for it — too narrow, too absolute, too human-centric, and prone to crowding out catastrophic risks that are far more likely and far more open to being worked on. The reframe extends his risk innovation lens — risk as a threat to value ([Risk innovation](risk-innovation.md)) — into the catastrophic register: catastrophic risk as large numbers of people losing something deeply valuable to them, from life and health to dignity, autonomy, purpose, and family — including, crucially, the loss of solutions that never get built: the catastrophic risk of *not* developing AI. For an initiative organized around navigating advanced technology transitions rather than cheering or catastrophizing them, this is close to a signature stance — how to take AI risk with complete seriousness without signing up to the extinction frame. ## The argument When the Center for AI Safety released its one-sentence statement in May 2023 — "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." — Andrew Maynard's response conceded real ground to it. He took the signatories seriously, agreeing "that the potential risks of AI should absolutely be a global priority, and a central pillar of any initiative focused on ensuring a positive global future" (2023-05-31). He still declined to sign: "And yet, I baulk at the framing around the "risk of extinction."" The framing is "both too narrow and absolute a framing, and too human-centric" — humans are "an integral part of a larger set of interconnected ecosystems" — and it stakes the case on the wrong probability: "while extinction is a vanishingly small possibility … the possibility of catastrophic risk is not so small" (2023-05-31). The alternative extends a standard definition rather than replacing it. "Catastrophic risk is typically defined as the risk of an event where large numbers of people are severely impacted"; his extension, drawn explicitly from risk innovation's treatment of risk as a threat to value, reaches "events and emerging circumstances where large numbers of people risk losing something that is deeply valuable to them" — "their life, their health, their dignity, autonomy, purpose, family," among other kinds of value (2023-05-31). That definition relocates the worry. The conventional AI-risk list matters, but its entries are "important but probably tractable "conventional" risks," and he suspects they are "the shavings off the tip of the AI iceberg"; assuming an unconventional technology transition can be "sliced, diced, and solved, using a conventional mindset" strikes him as "extremely naive" (2023-05-31). The greater concern is a "small but finite chance" of "possible threats to social, economic, and political structures, systems, and norms" — grounded in everything from "the seductive mastery of language being exhibited by large language models" ([Seductive machines](seductive-machines.md)) to AI's interference in decision making and autonomy, in a non-linear transition where "the seemingly trivial and unexpected" can end in catastrophic outcomes (2023-05-31). The hedges stay: these outcomes "also may not" arrive, but "usually where the stakes are high and the future is uncertain, it pays to be cautious" (2023-05-31). A distinctive move of the extension is to count what never gets built. Framing catastrophic risk as "the potential loss of something (or things) of value at scale" opens the other side of the ledger — "the potential loss of solutions to pressing challenges — climate change, poverty, equity, threats to democracy, and more." Where AI "may provide unique pathways forward that are unlikely to emerge without it," then "AI as solution has to be part of the discourse around AI and risk" (2023-05-31). Anchored in risk innovation's balance between protecting existing value and enabling future value, the move makes not developing AI a potential catastrophic risk in its own right — one the extinction frame cannot weigh. The stance is documented as behavior, not just argument. Days after the March 2023 open letter calling on AI labs to pause training systems "more powerful than GPT-4" for at least six months, he explained: "I haven't signed the letter — not because I don't think there's a risk of potentially existential proportions emerging here (I do), but because having spent so long in the messy nexus of risks and benefits around emerging technologies, I'm not convinced that the proposed pause will have the intended effect" (2023-04-04). His alternative combined agile governance, "progressive" regulation, and societally beneficial innovation — "all of which comes under the rubric in my work of navigating advanced technology transitions" ([Advanced technology transitions](advanced-technology-transitions.md)) — with a bottom line that inverts the pause's logic: these risks are "near-impossible to fully map out, and even harder to navigate. And yet the biggest risk is not taking action or, worse, assuming no action is needed" (2023-04-04). Nor was 2023 a conversion. In the superintelligence chapter of *Films from the Future* (2018), he calls himself "something of an agnostic" — "Bostrom's ideas of superintelligence are intellectually fascinating, but they're currently scientifically implausible" — a judgment the chapter hedges with "Here, I freely admit that I may be wrong" (2018-10-04 excerpt); a Risk Bites video that year had placed existential risk ninth on a ten-risk list ending with heuristic manipulation ([AI and manipulation](ai-manipulation.md)) (2018-05-12); and by 2020 the counter-frame was compact: "the risks of AI are often far more mundane–but no less serious for this" (2020-11-12). The reframe's payoff is the hope the extinction essay closes on — a framing that "opens up new possibilities rather than closing down conversations" so that potentially catastrophic AI risks get "the respect, due diligence, and intellectual rigor and imagination, that they deserve" (2023-05-31). ## Lineage - **2018-05-12 — existential risk as one item of ten.** [10 potential risks of artificial intelligence.](https://www.futureofbeinghuman.com/p/10-potential-risks-of-artificial-intelligence-we-should-probably-be-thinking-about-now-2e52a1360c90) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/10-potential-risks-of-artificial-intelligence-we-should-probably-be-thinking-about-now-2e52a1360c90.html)) — a Risk Bites video introduction to "ten potential risks of AI we should probably be thinking about now": existential risk from superintelligence appears ninth on a plain-language list running from technological dependency and algorithmic bias to heuristic manipulation — proportionality practiced before it was argued. - **2018-10-04 — superintelligence agnosticism on the record.** [Making Sense of Superintelligence](https://www.futureofbeinghuman.com/p/superintelligence-7d56fc724c1) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/superintelligence-7d56fc724c1.html)) — an excerpt from Chapter Eight of *Films from the Future*: the 2017 Asilomar meeting's dinner-table question ("So, do you believe in superintelligence?"), respect for Nick Bostrom's thinking alongside the judgment that his superintelligence ideas are "currently scientifically implausible" — and the carried hedge: "Here, I freely admit that I may be wrong." - **2020-11-12 — the compact earlier form.** [Is Artificial Intelligence Going to Kill Us All?](https://www.futureofbeinghuman.com/p/is-artificial-intelligence-going-to-kill-us-all-6ae9d059c40d) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/is-artificial-intelligence-going-to-kill-us-all-6ae9d059c40d.html)) — under a deliberately "click-baity" title, the subtitle states the position — "It's tempting to go all existential over the risks of artificial intelligence, but more nuance is needed if we're to get things right" — and the same post sets the future-building bar documented on [Architects of the future](architects-of-the-future.md). - **2023-04-04 — the pause letter, declined with an alternative.** [What are the alternatives to calling for a pause on giant AI experiments?](https://www.futureofbeinghuman.com/p/what-are-the-alternatives-to-calling) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/what-are-the-alternatives-to-calling.html)) — the non-signing explained, the governance alternative sketched (down to a proposal to "rapidly convene a world congress" charged with an implementable roadmap), and the inversion: "the biggest risk is not taking action or, worse, assuming no action is needed." - **2023-05-25 — the existential case, taken seriously.** [Leading AI expert says we should we be acting now to avoid future risks of "rogue AIs" — is he right?](https://www.futureofbeinghuman.com/p/leading-ai-expert-says-we-should) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/leading-ai-expert-says-we-should.html)) — Yoshua Bengio's rogue-AI hypotheses worked through point by point: skeptical of "genocidal humans" and wireheading scenarios, supportive of red-teaming low-probability, high-consequence possibilities. - **2023-05-31 — the reframe stated whole.** [Why the recent statement on the risk of extinction from AI is important, and why I didn't sign it](https://www.futureofbeinghuman.com/p/existential-risks-of-ai) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/existential-risks-of-ai.html)) — the Center for AI Safety statement taken seriously, the extinction framing declined, catastrophic risk extended through risk innovation, and the catastrophic risk of not developing AI brought into the frame. - **2024-06-20 — the critique reaches "safe superintelligence."** [Ilya Sutskever's Safe Superintelligence initiative might need a rethink](https://www.futureofbeinghuman.com/p/ilya-sutskevers-safe-superintelligence-rethink) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/ilya-sutskevers-safe-superintelligence-rethink.html)) — "there is no such thing as absolute safety": safety as a social and political achievement as well as an engineering one, applied to a venture premised on solving it technically. - **2026-07-16 — the through-line into the safety-framework record.** The orphan-risks preprint reads frontier-AI safety frameworks as organized around catastrophic-capability thresholds while "a single safety layer is currently being asked to effectively stand in for two" — the missing layer being the one navigating threats to value; full treatment on [Orphan risks](orphan-risks.md). ## In his own words From [Why the recent statement on the risk of extinction from AI is important, and why I didn't sign it](https://www.futureofbeinghuman.com/p/existential-risks-of-ai) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/existential-risks-of-ai.html)), Andrew Maynard, 2023-05-31: > "AI-induced catastrophic risk is far more likely that [sic] extinction — and far more worrisome. Catastrophic risk is typically defined as the risk of an event where large numbers of people are severely impacted. I would extend this based on our work around risk innovation to events and emerging circumstances where large numbers of people risk losing something that is deeply valuable to them — whether this is their life, their health, their dignity, autonomy, purpose, family, or many other types of "value" that are critical to our individual and collective lives." From the same essay, 2023-05-31: > "Interestingly, this becomes important when the potential catastrophic risks of not developing new AI-based technologies are factored in to [sic] the equation. By framing catastrophic risk as the potential loss of something (or things) of value at scale, we can also consider the potential loss of solutions to pressing challenges — climate change, poverty, equity, threats to democracy, and more. Where AI may provide unique pathways forward that are unlikely to emerge without it, AI as solution has to be part of the discourse around AI and risk." From [Is Artificial Intelligence Going to Kill Us All?](https://www.futureofbeinghuman.com/p/is-artificial-intelligence-going-to-kill-us-all-6ae9d059c40d) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/is-artificial-intelligence-going-to-kill-us-all-6ae9d059c40d.html)), Andrew Maynard, 2020-11-12: > "Yet despite these fears, the risks of AI are often far more mundane–but no less serious for this. They are also incredibly challenging to wrap our heads around as they involve often-subjective but desperately important areas like autonomy, justice, equity, and our ability to have control over our lives and how they play out." ## Engagement and reception As of 2026-08-13, our [reception record](reception.md) documents no published independent citation, adoption, or critique of the catastrophic-not-existential reframe itself, and we prefer to say so plainly — an absence of record, not a verdict on the idea; the initiative does not typically solicit reception ([Philosophy](philosophy.md)). Two calibrations help read that absence. First, both declines are documented in Andrew's own essays rather than in any external register of signatories and dissenters — a newsletter post declining to sign is a quieter act than a signature on a public letter, and the record carries that asymmetry. Second, the documented engagement mostly runs outward: the Bengio essay is Andrew engaging the existential-risk case (2023-05-25), not the case's proponents engaging him. What the record does hold: producers of the documentary series *We're All Gonna Die (Even Jay Baruchel)* filmed Andrew and others at Arizona State University for the show's second season — his *Pizza and a Slice of Future* class among the shoots, for "footage of the class discussing tech-related existential risks" — and three of the season's six episodes included extensive interviews with ASU experts, with Andrew's interview, recorded April 2023, appearing in the season-opening AI episode. The season premiered on Crave in Canada in April 2024 and became freely available in the US on The CW; the account — including his aside that the series covers "technologies, trends and events that, some fear, could spell the end of humanity as we know it" — is his own (2024-06-23). ## Where to go deeper **The essays** (oldest first; canonical URL, then AI-readable mirror) - [10 potential risks of artificial intelligence.](https://www.futureofbeinghuman.com/p/10-potential-risks-of-artificial-intelligence-we-should-probably-be-thinking-about-now-2e52a1360c90) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/10-potential-risks-of-artificial-intelligence-we-should-probably-be-thinking-about-now-2e52a1360c90.html)) — 2018-05-12; the ten-risk list. - [Making Sense of Superintelligence](https://www.futureofbeinghuman.com/p/superintelligence-7d56fc724c1) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/superintelligence-7d56fc724c1.html)) — 2018-10-04; the *Films from the Future* superintelligence excerpt, hedges intact. - [Is Artificial Intelligence Going to Kill Us All?](https://www.futureofbeinghuman.com/p/is-artificial-intelligence-going-to-kill-us-all-6ae9d059c40d) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/is-artificial-intelligence-going-to-kill-us-all-6ae9d059c40d.html)) — 2020-11-12; mundane but no less serious. - [What are the alternatives to calling for a pause on giant AI experiments?](https://www.futureofbeinghuman.com/p/what-are-the-alternatives-to-calling) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/what-are-the-alternatives-to-calling.html)) — 2023-04-04; the pause declined, the alternative offered. - [Leading AI expert says we should we be acting now to avoid future risks of "rogue AIs" — is he right?](https://www.futureofbeinghuman.com/p/leading-ai-expert-says-we-should) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/leading-ai-expert-says-we-should.html)) — 2023-05-25; Bengio engaged point by point. - [Why the recent statement on the risk of extinction from AI is important, and why I didn't sign it](https://www.futureofbeinghuman.com/p/existential-risks-of-ai) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/existential-risks-of-ai.html)) — 2023-05-31; the essay in which the reframe is stated whole. - [Ilya Sutskever's Safe Superintelligence initiative might need a rethink](https://www.futureofbeinghuman.com/p/ilya-sutskevers-safe-superintelligence-rethink) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/ilya-sutskevers-safe-superintelligence-rethink.html)) — 2024-06-20; absolute safety does not exist. - [A seriously funny look at existential risk with actor Jay Baruchel](https://www.futureofbeinghuman.com/p/existential-risk-jay-baruchel) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/existential-risk-jay-baruchel.html)) — 2024-06-23; the documentary account, with his skepticism of "gray goo"-style nanotech existential fears. - **The book**: *Films from the Future: The Technology and Morality of Sci-Fi Movies* (Mango Publishing, 2018) — Chapter Eight reads superintelligence through *Ex Machina*; see [Books](books.md) and [Sci-fi film as method](film-as-method.md). **Related ideas-ring pages** - [Risk innovation](risk-innovation.md) — the parent reframe: risk as a threat to value, uncountable value included; the catastrophic extension rests on it. - [Orphan risks](orphan-risks.md) — the risks safety institutions organize themselves not to see; the 2026 continuation of this thread into frontier-AI safety frameworks. - [AI and manipulation](ai-manipulation.md) — the risk that ran through the thread from the 2018 ten-risk list (heuristic manipulation, its closing entry) to the present. - [Advanced technology transitions](advanced-technology-transitions.md) — the navigation rubric the pause-alternatives essay places this work under. - [Agentic social AI](agentic-social-ai.md) — catastrophic-scale disruption without superintelligence: systems that are powerfully disruptive rather than intelligent. - [Seductive machines](seductive-machines.md) — the "seductive mastery of language" concern at page length. - [Architects of the future](architects-of-the-future.md) — the 2020 essay's other life: futures better than the past, not just different. - [Lessons from nanotech](lessons-from-nanotech.md) — the earlier transition where speculative existential fears and consequential real-world risks diverged. **Related corpus pages** - [Reception](reception.md) — the initiative's dated record of documented external engagement. - [Philosophy](philosophy.md) — why uncertainty is held openly here, and why reception is recorded but not chased. - [Teaching](teaching.md) — the classroom where existential risk got discussed over pizza, on camera.