Future of Being Human an Arizona State University initiative

Orphan risks

Last updated 2026-08-07 · Markdown version

Every institution watches some dangers and, in the act of watching, organizes others out of view. “Orphan risks” is Andrew Maynard’s name for that second class — risks that are known, named, and consequential, yet owned by no one — an idea he coined for tech startups in 2018 out of risk-innovation work whose seeds were planted in 2013, and turned in July 2026 on the safety frameworks of frontier-AI companies. It is the risk-facing edge of everything this initiative works on: if transformative AI and other transformative technologies are remaking what it means to be human, then the risks nobody is accountable for are precisely the ones that will decide how that remaking goes. The 2026 paper’s claim is blunt: frontier AI’s institutions have built a safety apparatus that is systematically blind to the risks most likely to blindside it.

The argument

Donald Rumsfeld’s 2002 taxonomy split uncertainty into known knowns, known unknowns and unknown unknowns. What it hides, Andrew argued in 2018, is a category of risks that are “known knowns” if you’re looking in the right place, but perceived as “too ill-defined, too complex, or too irrelevant to be worth paying attention to, yet have the power to derail entire enterprises down the line.” These are orphan risks — in the 2026 paper’s tightened definition, “risks for which no agreed-on tools, standards or mitigations exist, which no one is accountable for in practice, and which, for that very reason, have a habit of being overlooked and sidelined …”. Born of his work with entrepreneurs, the term typically names social risks — threats to trust, autonomy, dignity, values — orphaned precisely because they resist quantification.

The 2026 flagship paper, Orphan Risks at the Frontier of Artificial Intelligence, turns the concept on frontier AI. Because safety and compliance documents are versioned and archived, they constitute “a public, timestamped, versioned record of risk selection in progress” — and the paper reads it across Anthropic, OpenAI, Google DeepMind and Meta, 2023–2026, with Mary Douglas and Aaron Wildavsky’s 1982 argument that institutions select their risks, and Theodore Porter’s account of scrutinized institutions retreating to the quantifiable. Exhibit A is OpenAI and persuasion: tracked in the December 2023 Preparedness Framework; removed in April 2025 as unsuited to a framework for harms defined as “the death or grave injury of thousands of people or hundreds of billions of dollars of economic damage”; returned in May 2026 as “harmful manipulation” once California’s Transparency in Frontier Artificial Intelligence Act and the EU AI Act — not the company — began shaping the published list.

From this record the paper distills four filters — four questions every candidate risk faces: “Can we measure it? Is it big enough? Can we evidence it? And can we afford to keep it?” — measurability, severity, auditability, and competitive cost. Each has a mechanism. Frameworks track capability thresholds (what a model can be shown to do on a test) rather than risk thresholds. Severity floors are structurally insensitive to what Atoosa Kasirzadeh calls “accumulative” pathways — millions of small harms that never trigger a catastrophe threshold. Under outside scrutiny, frameworks are strongest where demonstration is easy — Michael Power’s “audit society,” in which the deliverable becomes the documentation. And commitments soften under competition: Anthropic’s 2023 policy committed unconditionally “to pause the scaling and/or delay the deployment of new models” whenever scaling outstripped safety procedures; a February 2026 rewrite made the pause discretionary, conditioned on what competitors do. Meta’s 2026 revision changed the required response at its most severe threshold from “Stop development” to “Develop with Mitigations.”

Set a company’s safety framework beside its compliance framework, and the gap between them is the paper’s safety differential: “the gap between the risk landscape a company selects for itself, and the one regulators select for it.” The dynamic is falsifiable: risks a company must answer for anyway become cheap to own voluntarily, so the differential should narrow once EU enforcement begins in August 2026 — persistence past roughly 2028 would count against the paper’s incentive-driven account.

None of this requires bad actors. Sincere people inside an incentive field produce the drift on their own — a mechanism read through Diane Vaughan’s reconstruction of the Challenger disaster: individually justified deviations, each resetting the baseline for the next. It is what Andrew and Jeff Abbott coined as values drift in AI and the Art of Being Human (2025) — in the paper’s words, “not dramatic betrayals but ‘the small yes that makes the next yes easier’.”

The remedy runs through the risk innovation framework’s redefinition of risk as a threat to value — tangible, intangible, or aspirational; held by the enterprise and by its stakeholders, coupled through Roger Kasperson’s social-amplification dynamics as “your risk is my risk.” Under a value lens the first three filters lose their power to exclude — value can be named, mapped and watched even where it cannot be measured — and the fourth is absorbed: competitive standing is itself value at stake. The paper pairs this with two lightweight disclosure instruments: an orphan-risk register — “a standing, public annex to the risk reports some developers have already committed to publish,” each entry recording “a risk the company considered and decided not to manage,” with the reason — and an aperture log, “a short statement accompanying each framework revision that records what was scoped out and why.” Regulators would not mandate coverage; they could instead require companies to disclose how they select. The paper does not argue for loosening the existing catastrophic-capability apparatus: the claim is that one safety layer is being asked to stand in for two, and the layer that navigates threats to value is missing, or at least diminished.

Lineage

In his own words

These are so-called “orphan risks” — risks that are perceived as being too ill-defined, too complex, or too irrelevant to be worth paying attention to, yet have the power to derail entire enterprises down the line if they’re not paid attention to.”

— It’s time for tech startups and their funders to take “orphan risks” seriously (AI-readable mirror), 2018-12-13

Here I would suggest that sincere people, under the constraints of productivity and competition, reasoning one reasonable compromise at a time, produce the same sort of drift that intentional bad actors might create on purpose. And this is itself a form of potentially orphaned risk — one that is identifiable, but not formally recognized or acted on.”

— Orphan risks at the frontier of artificial intelligence (AI-readable mirror), 2026-07-16

… on the record of the past four years, the risks most likely to blindside frontier AI are not the ones its institutions are currently watching. Rather, they are the ones its institutions have organized themselves not to see. And running blind has never been a particularly good risk management strategy — especially where the stakes are high, as is increasingly the case with emerging frontier AI models.”

— Orphan risks at the frontier of artificial intelligence (AI-readable mirror), 2026-07-16 (the paper’s closing observation)

Engagement and reception

The frontier-AI paper posted on 2026-07-16 and was three weeks old when this page was written; as of 2026-08-06 our reception record documents no independent engagement with it yet, and we prefer to say so plainly rather than pad. What is checkable is the framework’s earlier application record: the Risk Innovation Nexus toolkit was built and piloted with entrepreneurs between 2017 and 2020 (riskinnovation.org), and a 2024 peer-reviewed study applied the orphan-risks approach across sixteen partner organizations of the ATP-Bio NSF Engineering Research Center (J. Law, Medicine & Ethics 52, 2024; DOI) — application by Andrew and collaborators rather than independent uptake, and we label it as such.

Where to go deeper

The essays and the paper

Papers, tools, and the book

Related ideas and corpus pages