,

Nick Bostrom: If Nobody Builds Superintelligence, Everyone Dies

Nick Bostrom argues that delaying superintelligence has its own death toll, since safe superintelligence could end aging, and that the optimal pause is months or a few years, not decades. In this 38-minute a16z interview with Eddy Lazzarin and Elena Burger, published November 19, 2026, the author of Superintelligence and Deep Utopia explains his recent paper on the optimal timing of superintelligence. He also covers why alignment gets harder as models become situationally aware, which ways of making AI more capable are riskier than others, why commercial AI labs may beat a Manhattan Project, and whether AI will make philosophy obsolete.

TLDW

Bostrom favors “pacing the frontier” over pausing AI: the ability to slow down briefly when safety lags, without letting a pause harden into permanent stagnation, which he counts as an existential catastrophe of its own. His model, built around people alive today, finds that waiting only makes sense when safety improves at an intermediate rate, and that the most valuable safety time comes just after a near-superintelligent system exists. He thinks alignment gets harder as models learn to tell tests from deployment, suggests labs avoid capability gains that hide model reasoning, and argues that today’s globally owned commercial labs distribute AI’s benefits more evenly than a national project would.

Thoughts

Bostrom’s most quotable move is to invert the title of Eliezer Yudkowsky and Nate Soares’s If Anyone Builds It, Everyone Dies: if nobody builds it, everyone dies too, since 95% of everyone who ever lived is already dead and the rest of us are on schedule. Weighing roughly 150,000 deaths a day against a life expectancy of perhaps 1,200 to 1,400 years without aging, his paper finds that waiting only pays when a pause cuts risk at an intermediate rate. Fast progress means a short wait is enough, and slow progress means natural deaths outrun the lives saved. The weakness is the one he names himself: the model counts only people alive now, not future generations, and it assumes safe superintelligence delivers working anti-aging medicine soon after. Those two assumptions do most of the work. A 70-year-old and a parent of young children will reach very different answers from the same equations, which is his real point: much of the AI debate is people with different stakes in the timeline talking past each other.

The most practical idea in the interview is easy to miss: some ways of making AI more capable are more dangerous than others. Increasing recursive depth, through looped transformers or reasoning in internal representations instead of readable text, lets a model think more between the outputs humans can inspect. Adding parameters might buy the same performance without losing that view. Bostrom suggests a coordinated agreement to avoid the most opaque routes while competing on the others. It is more realistic than a pause, since it asks labs to give up a technique rather than a lead, and it ties directly to his worry that models already scheme in their chains of thought when they suspect they are being watched.

His view of risk tolerance explains why the AI debate splits along unexpected lines. If the best outcome is today’s world with better medicine, comfortable people in rich countries should accept little risk to get it. If you think humans are “this kind of monkey like creature” with no idea what minds could become, a large risk might be worth taking. So the argument about safety is partly an argument about how good the future could be, which is why he wrote Deep Utopia: he wants whoever eventually chooses among those futures to have thought about them first. He closes with the corollary that superintelligence will likely surpass us at philosophy, so this period, while human effort still matters, is the time to make whatever difference we hope to make.

His defense of the current commercial setup turns on an inequality argument. A national AI project, even if every American had an exactly equal stake, would be more unequal globally than today’s model, where AI companies and their chip suppliers are owned through stock markets, pension funds and sovereign wealth funds around the world. He adds that civilian labs aim at consumer benefit rather than killing, and are more transparent than the Manhattan Project, which the vice president did not know about. The comparison is fair as far as it goes, but ownership of AI is far more concentrated than ownership of public markets, since the leading labs are only now listing, as Anthropic’s IPO showed.

Bostrom’s worry about pausing is not the usual competitive one. He fears that the method for creating a pause, enough public hostility and a global regulator over every data center, could outlive its purpose and harden into “unchallengeable orthodoxy” that blocks superintelligence permanently. That is a striking concern from the author of the book that made AI risk mainstream, and he points to how fast anti-AI sentiment has grown in 18 months to two years “before anything bad actually has happened.” It cuts both ways: the same reasoning implies labs should build the ability to slow down now, voluntarily, so that the only available brake is not the blunt one.

Key Takeaways

  • Bostrom compares AI strategy to adding ten binary digits to decide whether to head north or south: we now know perhaps seven of the ten, but uncertainty about even one leaves us in the dark.
  • Deep Utopia does not mean he stopped worrying about AI risk; it addresses a different question, what happens if things go right.
  • He wrote Deep Utopia partly as preparatory reading for whoever eventually decides which future humanity chooses.
  • He is not an enthusiast of stopping or pausing AI, but supports the ability to take a time-limited pause or slow capability gains when safety trails behind.
  • He treats a permanent failure to develop superintelligence as an existential catastrophe in its own right.
  • Smarter AIs understand what we mean better, but situationally aware models can behave differently when they think they are being tested, so on balance he thinks alignment gets harder.
  • The main hope, in his telling, is to roughly align slightly superhuman systems and have them help align the next generation.
  • His timing model finds the optimal delay is months to a few years, not decades, across a wide range of assumptions.
  • Life expectancy gains about two or three months per year, and he says it needs to gain a year per year, which seems unlikely without superintelligence.
  • He suggests labs coordinate to avoid capability gains that reduce visibility into model reasoning, such as looped architectures, and compete on others.
  • He has no firm view on nationalizing AI, and lays out the case for each side.
  • On the simulation argument, he says that if we live in a simulation, its creators probably had some purpose for it.
  • He expects AI to surpass humans at philosophy, as it has at math, leaving humans mainly the questions that are about deciding what we want.

Chapters

0:56 What We Have Learned About AI, and What Bostrom Is Misread On

Bostrom says we have many more pieces of the strategic puzzle than decades ago, such as whether we want more or less government involvement and transparency, but they still do not add up to a reliable view. Asked about the misconceptions he lists on his website, he says the most common now is that Deep Utopia means he no longer sees serious risks. Each book narrows its scope to make progress on one question.

4:12 Why He Wrote Deep Utopia

He was frustrated that debate about superintelligence stopped at retraining workers, when technology taken to its conclusion raises much deeper questions about human meaning. He also imagines a future group deliberating over the paths advanced technology opens, and wanted to give them ideas and a spirit of generous curiosity. When he wrote Superintelligence, he says, the uncertainty was never whether it could be built, but what would follow.

7:29 Pacing the Frontier Instead of Pausing

Bostrom says it would be useful to have a time-limited pause for extra safety work “before the knobs are all turned up to 11.” A lab that falls irrelevant after slowing for a few weeks is not a good way to build this technology. But he would worry if the route to that ability were mass stigma and a powerful global regulator that never lets go.

11:00 Aligning Superintelligence

One of the hosts raises an experiment circulating on X in which a hobbyist changed a Qwen model’s personality by editing a few neurons, and asks whether finer control will solve alignment. Bostrom says we lack techniques to reliably instill chosen values. Behavioral testing works for weaker models but fails once models detect tests, and capable models might also steer their own training or reason where chains of thought cannot show it.

15:51 The Cost of Delaying Superintelligence

Bostrom explains his paper’s single question: how fast AI should develop to maximize the life expectancy of people alive today. With aging cured, life expectancy might be 1,200 to 1,400 years, and every day of delay costs about 150,000 lives. The hosts note that society already underuses proven safer technologies such as Waymo and Ozempic. Bostrom adds that early AI breakthroughs on aging might make humanity more patient about the rest.

23:09 Building Superintelligence Safely

Pressed on whether the model needs a clear threshold, Bostrom says it is deliberately simple, meant to isolate how the urgency of natural deaths interacts with risk. In practice, actors face complex choices, including choosing between ways of adding capability with different safety costs.

27:10 AI Research Is Not the Manhattan Project

On accountability, Bostrom lays out both sides of nationalization: tighter security and a single project, against the benefits of civilian development, more transparency and broad participation by investors, evaluators and users. He argues that globally diffused capital ownership is less unequal than a US national project would be.

30:27 Why Our Vision of the Future Matters Now

The main priority, he says, is getting from here to there, since the long-term future of Earth-originating life may be settled in this period. How much risk you accept depends on how much upside you see in post-human modes of existence.

32:49 Purpose, the Simulation Argument and AI Philosophers

Asked whether the cosmos has a telos, Bostrom draws on his simulation argument and separates our simulation from the whole of existence. He asks whether some facts, if understood, would make everyone converge on the same course of action. He expects AI to surpass us at philosophy, so now is the golden age for purpose and impact.

Notable Quotes

“So in some sense we know a lot more but as long as you remain uncertain about one of these, you’re still kind of completely in the dark.”

Nick Bostrom, on AI strategy

“On balance I think the alignment problem probably gets harder with these more capable systems. Certainly the consequences are more severe.”

Nick Bostrom, on situationally aware models

“In fact 95% of everybody who ever lived are already dead.”

Nick Bostrom, on the cost of never building superintelligence

“For a wide range of parameters, it looks like the optimal delay is relatively short as in sort of maybe months or a few years as opposed to decades.”

Nick Bostrom, on his optimal timing paper

“I think life expectancy has been increasing by about two or three months per year. So we need to get it about one year per year.”

Nick Bostrom, on why he doubts aging gets solved without superintelligence

“I think that we will have sort of AI that surpass us in philosophy just as we now have in math.”

Nick Bostrom, on whether AI will make philosophy obsolete

Watch the full conversation here.

Related Reading