However, his company will continue the development of AI.
Amodei, who drops these essays from time to time, suggests that humanity is about to enter a new era. “I believe we are entering a rite of passage, both turbulent and inevitable, that will test who we are as a species.” If things remain sideways this may even be our last era. “Humanity is about to be entrusted with almost unimaginable power, and it is deeply unclear whether our social, political and technological systems have the maturity to wield it,” Amodei said, later adding that “AI-enabled totalitarianism terrifies me.”
Side note: Anthropic offered the cloud to the Trump administration’s federal government for $1 per year.
To his credit, Amodei has a vivid imagination which he displays throughout the essay. He recalls the time when the Aum Shinrikyo religious movement released sarin nerve gas into the Tokyo subway in 1995, resulting in the death of 14 people and injuring many more. He then suggested that “putting a talent in everyone’s pocket” would remove the barrier to carrying out such attacks, or even deadly attacks.
He wrote, “The troubled loner who wants to kill people but lacks the discipline or skill to do so will now be elevated to the level of competence of a PhD virologist, who is unlikely to have this motivation.” “I’m concerned that there are potentially a large number of such people out there, and if they have an easy way to kill millions of people, sooner or later one of them will do it.”
Nothing, did you know that the assessment that Anthropic published in its “System Card” report for Cloud Opus 4.5 was a test where the model was tasked with helping virologists reconstruct a challenging virus?
Amodei is naturally impressed by the improvements we have seen in AI in recent years, but he warns that if improvements continue at this rate, we are not far from developing superintelligence – what people like Amodei used to call artificial general intelligence, but have since moved away from. He wrote, “If the exponential continues—which is not certain, but there is now a decade-long track record supporting it—then it may not take more than a few years for AI to be better than humans at essentially everything.”
What would this actually mean? Amodei offered an analogy: “Suppose that somewhere in the world in ~2027 a literal ‘nation of geniuses’ will come into existence. Imagine, say, 50 million people, all of whom are much more capable than any Nobel laureate, politician, or technologist,” he wrote. “Imagine further that, because AI systems can operate hundreds of times faster than humans, this ‘country’ is operating with a time advantage over all other countries: for every cognitive action we can take, this country can take ten.”
From that framework, Anthorpic’s CEO said it’s worth considering what our biggest concerns should be. Amodei proposed his own – including “autonomy risks,” “abuse to destroy,” and “abuse to seize power” – and ultimately concluded that the report on the country would consider it “the most serious national security threat in a century, possibly ever.”
A reminder that in anthropic analogy that country is being built.
Anthropic, more than any other AI firm, has been active in identifying the risks associated with AI development and advocating for additional regulatory scrutiny and consumer protections (whether you believe this is legitimate or a form of regulatory capture is in the eye of the beholder, but it’s at least talking a good game). But it continues to build a machine it warns may cause imminent destruction. You don’t have to build a doomsday machine! And frankly, continuing to build it undermines how seriously one should take warnings of existential threats.
If there is real concern that humanity may not be mature enough to handle AI, then maybe don’t make it publicly available to people with minimal barriers, and then brag about your monthly active users.
<a href