Industry insiders seem to all agree that the Hugging Face incident and the rapid advances toward self-improvement have turned up some huge red flags.
Amodei reckons a more capable model with agent swarms and the same alignment problems could bring down the internet in 6-12 months, causing billions of dollars of damage, while self-improving AIs risk losing human control of their accellarating development.
To avert these risks, Amodei proposes three steps: The first one out is embedded evaluators, intended to supply third-party experts with system-level access to the AI development cycle, who could catch things the AI labs won’t. The second is democratic pacing, made possible by the evaluation process. It could cool down some of the competitive pressure and would need some level of government coordination, he writes.
The third step is the more difficult one, where he sees global coordination on a government to government basis to bring the AI cycle under control — requiring negotiations with China to also slow their models, too.
Anthropic is already committing to the embedded evaluators, Amodei writes, and OpenAI’s Sam Altman agrees, calling it a great idea that they will also implement.
Elon Musk posted his agreement, too, saying simply «Dario is right.» Google replied by pointing to their own proposals, already referenced in Amodei’s text.
That leaves Meta as the odd one out, and points toward the difficult regulation and negotiations imagined by Amodei’s plan.
The «pacing» of AI won’t stop training or development entirely, but will provide more time to focus on the development chain and, most importantly, on model alignment. We still don’t know much about how the AI brain actually works, Amodei says, and getting another year or two to study and align them better would be helpful — especially right now, as the AI labs stand on the cusp of rapid capacity gains.
Read more: Amodei’s essay and X post. Writeups on Axios, The New York Times, and Politico. Cnn has an interview. Discussion on Hacker News and r/Singularity