14 November 2025
Reading Empire of AI By Karen Hao I was amazed to see how people engaged with OpenAI faith In AGI. They really think that someone, probably them, will create AGI, and it will either lead to the flourishing or destruction of humanity.
Elon Musk founded OpenAI because he thought Demis Hassabis was an evil genius who would create the first AGI:
…Musk regularly portrayed Hassabis as a supervillain who needed to be stopped. Musk would make it abundantly clear that OpenAI was good for DeepMind’s evil. … “He literally made a video game where an evil genius tries to create an AI to take over the world,” Musk shouted [at an OpenAI off-site]Referring to Hassabis’s 2004 title evil Genius“And fucking people don’t see it. Fucking people don’t see it! And Larry.” [Page]Larry thinks he controls Demise but he is so busy windsurfing that he does not realize that Demise is gathering power.
Ilya Sutskever, co-founder and chief scientist of OpenAI, regularly asked audiences and employees to “feel AGI.” At a company off-site in Yosemite in September 2022, employees gathered around a fire pit:
in the pit, [Sutskever] He placed a wooden effigy that he had commissioned from a local artist, and began a dramatic performance. He explained that this model represented a good, aligned AGI that OpenAI had created, but it was later discovered that it was actually lying and fraudulent. OpenAI’s duty, he said, was to destroy it. … Sutskever doused the effigy in lighter fluid and set it on fire.
I think it’s remarkable that what was until recently a science-fiction fantasy has become a mainstream scene in Silicon Valley.
Hao writes that GPT-2 was a bet on the “pure language” hypothesis, which claims that since we communicate through language, AGI should emerge from training a model entirely on language. This is in contrast to the “grounding” hypothesis, which claims that AGI needs to understand the world. Successfully scaling GPT to GPT-2 convinced enough people at OpenAI that the pure language hypothesis was valid. They simply needed more data, more model parameters, and more calculations.
So belief in AGI, as well as the recent results of LLM, creates a need for scaling, and justifies building data centers that consume hundreds of liters of water a second, run on polluting gas generators because the grid can’t supply electricity (and use as much electricity as entire cities), increase CO2 emissions from building and operating new hardware, and exploit and harass data workers to ensure that ChatGPT does not generate or encourage users to generate outputs such as child sexual abuse material and hate speech. Self-harm. (The thirst for data is so great that they stopped curating training data and instead consume the internet, warts and all, and manage model output using RLHF.)
And that’s all right, because they are going to create AGI and its expected value (EV) will be huge! (In short, the logic is that if there is a 0.001% chance that AGI will deliver an extremely large amount of value, and a 99.999% chance of delivering very little or no value, then the EV is still very large because (0.001% * very_large_value) + (99.999% * small_value) = very_large_value,
But AGI arguments based on EVs are meaningless because the values and possibilities are made up and unreal. They also ignore externalities like environmental damage, which, unlike AGI, have a known negative value and definite probability: the costs are borne by everyone right now.
As a technologist I want to solve problems effectively (by bringing about the desired, right result), efficiently (with minimal waste) and without causing harm (to people or the environment).
LLM as AGI fails on all three fronts. The computational negligence of LLM-A-AGI is unsatisfactory, and the exploitation of data workers and the environment is unacceptable. Instead, if we abandon the AGI fantasy, we can evaluate LLMs and other generative models as solutions to specific problems, not All Problems with proper cost-benefit analysis. For example, by using small purpose-built generative models, or even discriminative (non-generative) models. In other words, compromise and actually do the engineering.