Dwarkesh Podcast · Podcast summary

AI researchers debate how close we are to recursive self-improvement

AI researchers Baron Mollich, John Schulman, and Charlie O'Neill discuss the likelihood of achieving recursive self-improvement in AI, debating the technical and economic factors that could accelerate or hinder its arrival.

Dwarkesh Podcast · September 11, 2026 · 97-min episode · 5-min read

AI researchers debate how close we are to recursive self-improvement

Key takeaways

🧠Baron Mollich suggests that AI might hit a "Marvec's Paradox"-like wall, becoming extremely good at specific tasks but struggling with broader generalization and continual learning, preventing superintelligence.00:55
🔄John Schulman describes a recurring cycle where new AI models initially impress but then "start to feel dumb after a month or so" due to bottlenecks in areas requiring human-like judgment.01:48
💡Charlie O'Neill highlights that significant AI progress has often come from 'discontinuities' or new paradigms (like RLHF), and questions if current methods can discover the next necessary breakthrough.02:52
🎯Schulman firmly believes that defining objectives and deciding "what we actually want" will be the last job for humans, as it requires subjective judgment not easily automatable by AI.01:50:00
💧Distillation, the process of compressing knowledge from large models into smaller ones, is identified as a key factor preventing centralization in the AI model provider landscape.01:54:45
🇨🇳Chinese labs are already using "router services" to distill data from US frontier models, effectively democratizing access to advanced AI capabilities and leveling the playing field.02:50:00

Chapters

0:00🎙️ Introduction to the Panel and Topic
0:29👽 The Likelihood of a 'Crazy Alien Superintelligence World' by 2036
2:30🎯 The Last Human Job: Defining What We Want
3:09📊 The Impact of Data and Distillation on Centralization
3:35🤖 Training the First AI R&D Automators
5:07💡 The Role of Discontinuities and Defining Objectives
5:07🔮 The Future of AI: A Long Tail of Challenges

The digest

In a recent podcast episode, AI researchers Baron Mollich (CTO of Zyphra), John Schulman (Chief Scientist at Thinking Machines, co-founder of OpenAI, and lead of RLHF work), and Charlie O'Neill (Head of Model Training at Base10) engaged in a lively discussion about the future of AI, specifically focusing on the timeline and feasibility of achieving recursive self-improvement (RSI).

The Likelihood of a 'Crazy Alien Superintelligence World' by 2036

The conversation kicked off with a provocative question: if it's 2036 and we haven't seen a world transformed by billions of superintelligent AIs, what's the most likely technical reason? Baron Mollich (00:55) proposed that the AI might hit a "Marvec's Paradox"-like wall, becoming "extremely good at kind of everything that people like put into a benchmark... but like there are still some persistent like sim-to-real mismatches somehow blocking everything." He suggested that if generalization and continual learning remain ridiculously hard, AI might plateau before reaching superintelligence.

John Schulman (01:48) echoed this, describing a cycle where new models initially impress but then "start to feel dumb after a month or so." He emphasized that current models are often bottlenecked by their weaker areas, and even if they can write more code than a person, they don't make humans 100 times more productive. Charlie O'Neill (02:52) framed it as a question of how far off the "global optimum of a learner" current models are. He highlighted that while current methods like self-attention and RL on scaling environments have shown impressive progress, this has often been driven by "discontinuities" or new paradigms (like RLHF after diminishing returns in pre-training). He questioned if current methods could discover the next necessary discontinuity, suggesting that without it, AI might hit an "asymptotic curve."