Future of Being Human an Arizona State University initiative

Catastrophic, not existential

Last updated 2026-08-13 · build 20260814T0314Z-83bb462d · Markdown version · corpus index

In spring 2023, two widely signed public declarations set the terms of the AI-risk debate: a March open letter calling for a pause on giant AI experiments, and a May statement from the Center for AI Safety that mitigating “the risk of extinction from AI” should be a global priority. Andrew Maynard declined to sign either — and published a reasoned alternative to each. His position, built on a career in risk science running from aerosols through nanotechnology to transformative AI and other transformative technologies, holds two commitments the 2023 debate treated as opposites: AI risk genuinely belongs among the world’s priorities, and the extinction framing is the wrong vehicle for it — too narrow, too absolute, too human-centric, and prone to crowding out catastrophic risks that are far more likely and far more open to being worked on. The reframe extends his risk innovation lens — risk as a threat to value (Risk innovation) — into the catastrophic register: catastrophic risk as large numbers of people losing something deeply valuable to them, from life and health to dignity, autonomy, purpose, and family — including, crucially, the loss of solutions that never get built: the catastrophic risk of not developing AI. For an initiative organized around navigating advanced technology transitions rather than cheering or catastrophizing them, this is close to a signature stance — how to take AI risk with complete seriousness without signing up to the extinction frame.

The argument

When the Center for AI Safety released its one-sentence statement in May 2023 — “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” — Andrew Maynard’s response conceded real ground to it. He took the signatories seriously, agreeing “that the potential risks of AI should absolutely be a global priority, and a central pillar of any initiative focused on ensuring a positive global future” (2023-05-31). He still declined to sign: “And yet, I baulk at the framing around the “risk of extinction.”” The framing is “both too narrow and absolute a framing, and too human-centric” — humans are “an integral part of a larger set of interconnected ecosystems” — and it stakes the case on the wrong probability: “while extinction is a vanishingly small possibility … the possibility of catastrophic risk is not so small” (2023-05-31).

The alternative extends a standard definition rather than replacing it. “Catastrophic risk is typically defined as the risk of an event where large numbers of people are severely impacted”; his extension, drawn explicitly from risk innovation’s treatment of risk as a threat to value, reaches “events and emerging circumstances where large numbers of people risk losing something that is deeply valuable to them” — “their life, their health, their dignity, autonomy, purpose, family,” among other kinds of value (2023-05-31).

That definition relocates the worry. The conventional AI-risk list matters, but its entries are “important but probably tractable “conventional” risks,” and he suspects they are “the shavings off the tip of the AI iceberg”; assuming an unconventional technology transition can be “sliced, diced, and solved, using a conventional mindset” strikes him as “extremely naive” (2023-05-31). The greater concern is a “small but finite chance” of “possible threats to social, economic, and political structures, systems, and norms” — grounded in everything from “the seductive mastery of language being exhibited by large language models” (Seductive machines) to AI’s interference in decision making and autonomy, in a non-linear transition where “the seemingly trivial and unexpected” can end in catastrophic outcomes (2023-05-31). The hedges stay: these outcomes “also may not” arrive, but “usually where the stakes are high and the future is uncertain, it pays to be cautious” (2023-05-31).

A distinctive move of the extension is to count what never gets built. Framing catastrophic risk as “the potential loss of something (or things) of value at scale” opens the other side of the ledger — “the potential loss of solutions to pressing challenges — climate change, poverty, equity, threats to democracy, and more.” Where AI “may provide unique pathways forward that are unlikely to emerge without it,” then “AI as solution has to be part of the discourse around AI and risk” (2023-05-31). Anchored in risk innovation’s balance between protecting existing value and enabling future value, the move makes not developing AI a potential catastrophic risk in its own right — one the extinction frame cannot weigh.

The stance is documented as behavior, not just argument. Days after the March 2023 open letter calling on AI labs to pause training systems “more powerful than GPT-4” for at least six months, he explained: “I haven’t signed the letter — not because I don’t think there’s a risk of potentially existential proportions emerging here (I do), but because having spent so long in the messy nexus of risks and benefits around emerging technologies, I’m not convinced that the proposed pause will have the intended effect” (2023-04-04). His alternative combined agile governance, “progressive” regulation, and societally beneficial innovation — “all of which comes under the rubric in my work of navigating advanced technology transitions” (Advanced technology transitions) — with a bottom line that inverts the pause’s logic: these risks are “near-impossible to fully map out, and even harder to navigate. And yet the biggest risk is not taking action or, worse, assuming no action is needed” (2023-04-04).

Nor was 2023 a conversion. In the superintelligence chapter of Films from the Future (2018), he calls himself “something of an agnostic” — “Bostrom’s ideas of superintelligence are intellectually fascinating, but they’re currently scientifically implausible” — a judgment the chapter hedges with “Here, I freely admit that I may be wrong” (2018-10-04 excerpt); a Risk Bites video that year had placed existential risk ninth on a ten-risk list ending with heuristic manipulation (AI and manipulation) (2018-05-12); and by 2020 the counter-frame was compact: “the risks of AI are often far more mundane–but no less serious for this” (2020-11-12). The reframe’s payoff is the hope the extinction essay closes on — a framing that “opens up new possibilities rather than closing down conversations” so that potentially catastrophic AI risks get “the respect, due diligence, and intellectual rigor and imagination, that they deserve” (2023-05-31).

Lineage

In his own words

From Why the recent statement on the risk of extinction from AI is important, and why I didn’t sign it (AI-readable mirror), Andrew Maynard, 2023-05-31:

“AI-induced catastrophic risk is far more likely that [sic] extinction — and far more worrisome. Catastrophic risk is typically defined as the risk of an event where large numbers of people are severely impacted. I would extend this based on our work around risk innovation to events and emerging circumstances where large numbers of people risk losing something that is deeply valuable to them — whether this is their life, their health, their dignity, autonomy, purpose, family, or many other types of “value” that are critical to our individual and collective lives.”

From the same essay, 2023-05-31:

“Interestingly, this becomes important when the potential catastrophic risks of not developing new AI-based technologies are factored in to [sic] the equation. By framing catastrophic risk as the potential loss of something (or things) of value at scale, we can also consider the potential loss of solutions to pressing challenges — climate change, poverty, equity, threats to democracy, and more. Where AI may provide unique pathways forward that are unlikely to emerge without it, AI as solution has to be part of the discourse around AI and risk.”

From Is Artificial Intelligence Going to Kill Us All? (AI-readable mirror), Andrew Maynard, 2020-11-12:

“Yet despite these fears, the risks of AI are often far more mundane–but no less serious for this. They are also incredibly challenging to wrap our heads around as they involve often-subjective but desperately important areas like autonomy, justice, equity, and our ability to have control over our lives and how they play out.”

Engagement and reception

As of 2026-08-13, our reception record documents no published independent citation, adoption, or critique of the catastrophic-not-existential reframe itself, and we prefer to say so plainly — an absence of record, not a verdict on the idea; the initiative does not typically solicit reception (Philosophy). Two calibrations help read that absence. First, both declines are documented in Andrew’s own essays rather than in any external register of signatories and dissenters — a newsletter post declining to sign is a quieter act than a signature on a public letter, and the record carries that asymmetry. Second, the documented engagement mostly runs outward: the Bengio essay is Andrew engaging the existential-risk case (2023-05-25), not the case’s proponents engaging him. What the record does hold: producers of the documentary series We’re All Gonna Die (Even Jay Baruchel) filmed Andrew and others at Arizona State University for the show’s second season — his Pizza and a Slice of Future class among the shoots, for “footage of the class discussing tech-related existential risks” — and three of the season’s six episodes included extensive interviews with ASU experts, with Andrew’s interview, recorded April 2023, appearing in the season-opening AI episode. The season premiered on Crave in Canada in April 2024 and became freely available in the US on The CW; the account — including his aside that the series covers “technologies, trends and events that, some fear, could spell the end of humanity as we know it” — is his own (2024-06-23).

Where to go deeper

The essays (oldest first; canonical URL, then AI-readable mirror)

Related ideas-ring pages

Related corpus pages