Dario's (likely half-sincere) AI Slowdown proposal ... a hard no from me and from reality

The conversation around existential risk from Artificial General Intelligence (AGI) has abruptly moved from the fringes of academic thought to the forefront of public discourse. For those who have been immersed in the AGI field for years, this moment feels both inevitable and overdue. The foundational arguments, largely popularized by figures like Nick Bostrom in his book "Superintelligence" (2012), which itself built upon the work of Eliezer Yudkowsky, initially sparked brief alarm among tech titans like Elon Musk and Bill Gates. However, this initial concern often quickly pivoted, as seen with Musk, towards a drive to build and control AGI rather than to halt its development.

The Evolution of AI Safety Narratives

Fast forward to today, and while human-level AGI remains elusive, the rapid advancements in AI technology clearly indicate we are on a path towards it. The renewed urgency in the AI safety debate is marked by a more nuanced, albeit still often flawed, set of concerns. Eliezer Yudkowsky's recent assertion that "if anyone builds it, we all die" represents an extreme end of this spectrum, positing an almost inevitable doom from AGI unless it is built with impossibly strong, provable safety guarantees. This perspective, however, is critiqued for its inherent impracticality, as a formal model of "life, the universe, and everything" – the complex environment AGI would interact with – is beyond our current grasp. Even with a robust formal model of AGI itself (which is a goal of projects like Hyperon), the inability to formalize the entire universe precludes absolute safety guarantees.