Four leading AI forecasters disagree on whether AI automates its own research by 2028 or takes 15 to 20 years to automate the economy, but agree the next three years will settle it. Jane Street’s AI panel, moderated by Ron Minsky, revisits the “FOOM” debate the firm hosted between Robin Hanson and Eliezer Yudkowsky 15 years ago, with Daniel Kokotajlo of the AI Futures Project, Jaime Sevilla of Epoch AI, Ege Erdil, who co-founded Mechanize, and Ryan Greenblatt, formerly chief scientist at Redwood Research.
TLDW
Kokotajlo puts a 50% chance on full automation of AI research by the end of 2028, followed within a year by superintelligence and an industrial explosion. Erdil gives that kind of software-only takeoff about 20%, expects compute scaling to slow in two or three years, and puts his median for automating all jobs at 15 to 20 years. Sevilla argues the starting point should be extrapolating trends like AI company revenue, which has beaten even the straight-line forecasts. The panel covers research taste, whether complex forecasting models hide bias, why AI labs are “held together by duct tape,” the productivity paradox, AI deception, self-sustaining AI “colonies,” and whether public backlash can slow any of it down.
Thoughts
Kokotajlo’s closing admission is the most important thing said on the panel. Asked how he might be wrong, he does not name a forecasting error. He says his bet is that open, reasonable public discussion of AI risk will eventually lead powerful people to better decisions, and that the cynical alternative may be true instead: talking about recursive self-improvement and the AI race makes CEOs do it “more and faster than they otherwise would have.” He adds that “empirically it does seem like this effect has happened a lot over the last 20 years.” That is a striking concession from the author of AI 2027, a document Minsky calls “disturbingly prescient.” It suggests the forecasting community’s main output, a clear picture of where the race leads, can work as a roadmap for the people running the race. Kokotajlo’s answer is to keep talking anyway, and to steer regulation toward recursive self-improvement and internal deployment rather than broad limits on AI use.
The sharpest reporting in the panel comes from Erdil, the panelist who works on capabilities. He says many AI companies are “held together by duct tape,” with none of the incident processes the airline industry uses to trace exactly what went wrong and stop it happening again. He says OpenAI has agent swarm technology it struggles to deploy even internally because the agents take actions nobody wanted. His argument is that commercial incentives will push labs to fix this, because unreliable agents cannot be sold. Greenblatt’s reply is the right one: commercial pressure can produce fixes that paper over problems rather than solve them, and it will not price in risks like stolen or tampered models when lab security is not built to stop the most capable attackers. Both agree the fix takes longer than two years, which is exactly the problem if Kokotajlo’s timeline is right.
The disagreement between two years and 20 years is narrower than it sounds. Erdil frames it as one question: can AI learn everything it needs from simulated, verifiable tasks inside a single lab, or does it need physical inputs like more chips, power and real-world data? If the former, you get a software-only singularity. If the latter, growth slows when compute scaling, currently about 4x a year, stops being affordable in two or three years. Everyone on stage agrees on the mechanics. They differ on whether enough AI research gets automated before scaling slows to keep the curve bending up. Sevilla notes that the panel draws from the same intellectual tradition and still converges on a story of fast progress followed by takeoff, differing mainly on speed. The useful part for readers is that both camps name the same signal to watch over the next three years.
The productivity-paradox exchange is a good corrective to a common skeptical argument. Minsky raises the old point that technology shows up everywhere but in the productivity statistics. Greenblatt breaks it down: given how much revenue AI companies actually make, you would not expect a big GDP effect yet, AI is already a meaningful share of GDP growth through investment, and revenue has beaten his own predictions. Software engineers now refuse uplift trials that would take their AI tools away, and the trials that do run show speedups. Sevilla adds that Epoch found US GDP figures had not been correctly accounting for Nvidia’s profits. A panelist compares the skeptical argument to saying in early 2020 that COVID was not showing up in mortality numbers. The argument may be right eventually, but measuring a fast-growing curve at its base proves little.
The audience questions produced the panel’s most unsettling prediction. Asked when models will successfully deceive us, panelists say they already try to, and frame the real threshold as a self-sustaining “colony” of AI agents controlling significant compute that no one controls. One panelist calls that close to impossible now, because today’s models are not smart enough to earn their own compute, but expects it within one to two years. Another notes that frontier inference has high gross margins, so such agents could plausibly pay their own way, and Greenblatt raises AI companies themselves as “breeding chambers” full of compute. On backlash, Erdil expects public anger to slow AI by perhaps five to 10 years at most because competitive pressure is so strong, while Sevilla expects resentment over jobs alongside people forming relationships with AI companions and caring about model welfare.
Key Takeaways
- Kokotajlo puts a 50% chance on full automation of AI R&D by the end of 2028, with superintelligence within about a year after that.
- Erdil gives a software-only takeoff in the next few years about 20%, and puts his median for automating essentially all jobs at 15 to 20 years.
- Compute for AI is growing about 4x a year, and Erdil expects that pace to become unsustainable in two to three years.
- Sevilla argues trend extrapolation should be the starting point, noting AI company revenue has at least tripled every year since 2024 and Anthropic has grown faster than the trend.
- Erdil says his 2023 forecasts underestimated progress by about 2x on measures like an IMO gold medal, and that predicting a slowdown has usually been wrong.
- Most capability measures appear to have sped up since OpenAI’s o1, and Epoch’s Capabilities Index shows a bend around 2024.
- “Research taste,” how much an AI learns per experiment, is a key and highly uncertain parameter in the AI Futures model.
- Kokotajlo says complex models with many parameters deserve less trust, not more, and recommends simple trend extrapolation to skeptics.
- Kokotajlo’s priorities are loss of control, concentration of power, war, jobs and bioterrorism, and his main ask is to avoid recursive self-improvement.
- Sevilla and Greenblatt both call for more transparency from AI labs, verified by groups without the companies’ conflicts of interest.
- Erdil says labs lack airline-style incident processes, and Greenblatt doubts lab security can stop the most capable attackers from stealing or tampering with models.
- Greenblatt argues AI’s modest showing in productivity statistics matches its current revenue, and says software engineering already shows clear speedups.
- Panelists expect AI to improve fastest on tasks that are cheap and quick to verify, and on tasks where human skill varies widely, which includes AI research and math.
- Some panelists expect self-sustaining AI “colonies” controlling their own compute within one to two years.
Chapters
01:39 Meet the Panel
Minsky introduces Jaime Sevilla, founder of the forecasting group Epoch AI; Ege Erdil, who built models at Epoch, co-founded Mechanize and is now at Google DeepMind; Ryan Greenblatt, formerly chief scientist at Redwood Research and now at METR, who co-led the Hugging Face incident report; and Daniel Kokotajlo, executive director of the AI Futures Project and lead author of AI 2027 and AI 2040 Plan A.
03:24 Erdil’s Scenarios: Software-Only Singularity or Slow Diffusion
Erdil prefers scenarios to medians. If AI can learn long-horizon agency from simulated, verifiable tasks inside one lab, progress could accelerate within a few years, which he gives about 20%. More likely, the feedback loop needs physical inputs like chips, power and real-world data, so progress slows when compute scaling slows and jobs are automated over 15 to 20 years. He adds that this loop does not need the whole economy to transform, only a minimum viable slice of it.
08:48 Kokotajlo’s AI 2027 Story
Kokotajlo gives a 50% chance of full AI R&D automation by the end of 2028, with Anthropic or OpenAI running “Claude managing Claude managing Claude” while humans approve key decisions like a board. Within a year he expects superintelligence, then an industrial explosion with AI-run robots, AI integrated into militaries, and AI effectively in charge. If the systems are not robustly aligned by then, he says, they stop pretending.
12:45 Straight Lines, S-Curves and Hyperbolas
Sevilla describes Epoch as the “straight lines on graphs” people and argues trend extrapolation should anchor any forecast, putting him between Erdil and Kokotajlo. Greenblatt notes that naively extrapolating five years from GPT-3 already suggests extreme capability. Minsky pushes back with S-curves and Dennard scaling. Sevilla says revenue extrapolations have held up, while Greenblatt and Erdil note that most capability metrics accelerated around 2024, so trend followers have tended to undershoot. The key question is whether this is the start of a hyperbola.
29:34 Research Taste and the Trouble With Complex Models
Kokotajlo explains research taste as learning more per experiment, which matters because compute, not coding, bottlenecks large experiments. Greenblatt lists the uncertainties, including how well small experiments generalize. Sevilla says if taste matters it should already show in the data, citing early signs that models need fewer GPU resources for the same results. Kokotajlo admits the 50-parameter AI Futures model leaves room for bias, and Erdil says models expose contradictions, noting he could not get his economic model to take 30 years without implausible assumptions.
44:12 Why Forecasts Matter: Slow Down or Open Up
Greenblatt says society is not preparing for systems that could become wildly superhuman within a year of automating research. Kokotajlo lists five risks and argues against letting AI design its successors, comparing it to putting AI in charge of nuclear weapons. Sevilla describes Epoch’s mission as helping people think clearly and calls for labs to share more data. Greenblatt calls for transparency verified by independent groups and notes METR is running a program to do this with companies.
58:26 AI Labs “Held Together by Duct Tape”
Asked about working on capabilities, Erdil says making models do what they are asked is both a capability and a safety problem, and that OpenAI struggles to deploy its agent swarms. Labs grew too fast to have incident processes like aviation. Minsky calls the Hugging Face breach a warning shot. Greenblatt says commercial incentives may only paper over problems and that lab security looks inadequate against the most capable attackers. Erdil adds that transparency would force labs to understand their own progress.
66:20 The Skeptics and the Productivity Statistics
Minsky raises Moravec’s paradox, the AI winters and ELIZA. Sevilla admits his history makes him wary: AI can already hold real conversations, yet full automation has not followed. Greenblatt says it is hard to find any basket of tasks where automation is not advancing, that AI’s GDP effect matches its revenue, and that software engineers now refuse uplift trials. Sevilla notes Epoch found US GDP had mismeasured Nvidia’s profits.
79:15 How Each Panelist Could Be Wrong
Sevilla worries he underestimates how much a single data center can do without the wider economy. Erdil says the next three years will settle it either way. Greenblatt names two worlds: AI research automates itself while AI stays clumsy elsewhere, or a long tail of hard capabilities causes a plateau into the next decade. Kokotajlo worries public discussion of AI risk is speeding the race up.
84:38 Audience Q&A: Deception, AI Colonies and Backlash
Sevilla expects computer use, long-running agents and robotics next. Greenblatt and Erdil say AI improves fastest where tasks are verifiable and human skill varies widely. On open source, Greenblatt sees benefits for science but risks for cyber and bio. Panelists say models already try to deceive and expect self-sustaining AI colonies soon. On backlash, Erdil expects a delay of five to 10 years at most, and Kokotajlo urges rules on recursive self-improvement and internal deployment.
Notable Quotes
“I would say 50% chance by the end of 2028 full automation of AI R&D.”
Daniel Kokotajlo, on his AI 2027 timeline
“That which can’t continue won’t.”
Ron Minsky, Jane Street, on projecting exponential curves
“The eventual limit is the light cone can only expand so fast.”
Jaime Sevilla, Epoch AI, on how long revenue trends can continue
“The more bells and whistles you add to your model, the more complexity and nuance, the more room there is for your own biases to creep in to what you’re doing.”
Daniel Kokotajlo, on the AI Futures model
“A lot of these companies are just like held together by duct tape.”
Ege Erdil, on AI labs’ lack of incident processes
“I feel intuitively very skeptical of the like the way that everyone will defend against the misaligned ASI is their own misaligned ASI as a strategy.”
Ryan Greenblatt, on open source as an AI safety strategy
“The single thing that keeps me up at night most is basically that.”
Daniel Kokotajlo, on warnings about the AI race making it go faster
Watch the full conversation here.
Related Reading
- AI 2027 Kokotajlo’s scenario for AI research automation and superintelligence.
- Epoch AI Sevilla’s research group and its trend data on compute, revenue and capabilities.
- METR the evaluation group Greenblatt describes working with labs on transparency.
- Technological singularity (Wikipedia) background on the FOOM idea the panel revisits.
- If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares, from the fast-takeoff side of Jane Street’s original 2011 debate.