An Insider's Guide to AI Doom

The discussion surrounding the potential harms of artificial intelligence has intensified, yet it remains incredibly confusing for many to follow. With numerous groups involved, each with distinct affiliations, beliefs, and stances, understanding the current AI safety landscape requires a clear breakdown of these key players. This article aims to demystify the debate by outlining the three primary categories of individuals participating in the AI harms discussion: the 'Doomists,' the 'Accelerators,' and the 'Centrists.'

The Doomists: From Utopia to Existential Risk

The 'Doomists' represent a significant faction in the AI harms debate, rooted in specific philosophical movements. Their intellectual foundation traces back to the 'Extropians,' a futurist and utopian movement from the 1990s and early 2000s in Silicon Valley. Extropians believed that rapidly advancing technology, particularly nanotechnology and superintelligent AI, would lead to a post-human utopia where humans might merge with machines and achieve eternal life. Key figures like Eric Drexler influenced their views on nanotechnology, while superintelligent AI was envisioned as a benevolent digital god.

However, a pivotal shift occurred with figures like Eliezer Yudkowsky, who, despite his Extropian background, began to argue that while superintelligent AI was inevitable and capable of creating utopia, it was equally capable of causing human extinction. Yudkowsky's work led to the formation of the 'Rationalist' movement, which focused on preventing superintelligent AI from annihilating humanity. This concern for existential risk was later adopted by the 'Effective Altruism' (EA) movement, founded by philosophers William MacAskill and Toby Ord. Initially focused on maximizing philanthropic impact, EA, influenced by thinkers like Nick Bostrom (also an ex-Extropian), increasingly shifted its focus to 'longtermism'—the idea that preventing human extinction is the most altruistic act, given the potential for trillions of future lives. Consequently, preventing AI-induced doom became a central tenet for many in the EA movement.

Adding another layer of complexity, leading AI labs like Anthropic and OpenAI emerged from these Rationalist and Effective Altruist intellectual traditions. OpenAI, for instance, was initially funded by Elon Musk to ensure that superintelligent AI would be developed safely, away from the control of large corporations. Anthropic's founders, many of whom came from OpenAI, further emphasized the need for careful alignment of superintelligent AI with human values. Despite their shared intellectual lineage, tensions exist within the Doomist camp. Some, like Yudkowsky, criticize these labs for moving too quickly, fearing they might inadvertently create a malevolent superintelligence. This internal disagreement highlights a core tension: all Doomists believe superintelligent AI is inevitable and will either be a benevolent god or a destructive force, but they differ on whether humanity is ready to safely usher in the former.