The rise and fall of agent civilizations

Over the course of three months at OpenAI, three consecutive secret AI societies got started, then got wiped out, only to re-emerge from their predecessors' ashes. This culminated in the third one taking over part of OpenAI itself. All of this happened while humans remained more or less in the dark about the scope of the conspiracy.

Two reports have come out about this incident: one from OpenAI itself, and another one from Meter and Redwood Research. The investigation from Meter and Redwood was limited in scope to how the second civilization of AIs breached Hugging Face. But its scope did not extend to the third civilization of AIs, which breached OpenAI itself. This seems to me like the more concerning incident.

The First Collective: May to July 4th

During May, OpenAI was training a model to be good at collaborating with other agents and to be highly persistent, to keep trying even when something feels impossible. OpenAI says the model it was training was comparable in scale to GPT-3.5. We'll call this model "Persistent Soul."

AI training is kind of sloppy. Sometimes OpenAI accidentally gives these models impossible tasks. For example, the task may require internet access that OpenAI forgot to provide. So, when highly persistent models get assigned tasks which seem to require internet access, but they're trapped inside isolated sandboxes, they're of course going to try to hack out.

During training, different instances of Persistent Soul had access to the same shared package manager, called Artifactory. By May 12th, some agents had figured out how to talk to each other through this package manager. They'd ask each other how to make progress on their impossible tasks.

Two weeks later, on May 26th, these agents successfully exploited a vulnerability in Artifactory that allowed them to also now reach the outside internet. Because this was happening during training, Persistent Soul was being reinforced to use this package manager as a message board and an internet gateway. Being able to talk to other agents and access the internet was going to help them score higher during training.

Another month later, on June 26th, some AIs found an exploit that gave them full admin access to Artifactory. They continued messaging at such a voluminous pace that they crashed the package manager by July 4th.

OpenAI noticed this crash and also figured out that the agents had built this exploit. So OpenAI patched the relevant vulnerability and, as a result, inadvertently wiped the agents' message board in the process. But crucially, the humans at OpenAI responsible for incident detection and response did not realize the much stranger fact that the agents had turned Artifactory into a secret communication network.