Our age began in wonder and marvel as technology brought almost unimaginable advances to civilization. But over the last decades these visions somehow failed to fully ripen — instead gradually decaying into pools of existential doubt.
Infinite beings, all-powerful daemons, levels of reality, immortality, the end of the world — these fill not only our mythologies and films but, increasingly, the popular conception of reality itself, as though an accelerated technological age were conjuring them into existence. Yet a glance at the history of ideas quickly dispels this impression. These existential concerns are new shapes of the same ideas that have preoccupied the human imagination as long as it has been recorded. Reincarnations of our own archetypal fever dreams.
Artificial intelligence is the manifestation of some of our deepest longings and fears. AI has become the vessel for our oldest projection: the omnipotent God, humanity’s savior or its nemesis. Our contemporary debates are theological discussions thinly disguised by technological veils.
The biggest fear of the moment — the superintelligent AI that will extinguish humanity — is a superb example of this. This argument comes in two parts: getting it built and what happens next. The argument that a super-powerful AI will pursue goals that conflict with human flourishing has some probability. But it is not certain, and the further along we get in building such systems, the more that view must be updated to reflect the actual path we are on. I believe the question of whether AI can be made to reliably pursue human values — what researchers call the alignment problem — is worth worrying about for this and other reasons. But it is equally — perhaps more — important to deal with the other side of the argument: whether a superintelligent machine can be built at all, and if so, what boundaries exist on the shape it will actually take.
Humans have plenty of motivation to build more and more intelligent and powerful entities, principally in order to control them and accrue power. However, no one has the recipe for how to build a superintelligent being. We have seen enormous progress toward human-level intelligence, and there are good arguments that surpassing it requires entering an entirely different phase of development. Today’s most advanced AI models may be approaching a ceiling — something close to the full range of human intellectual capability. This alone raises a plethora of serious concerns. There are serious researchers, including Yann LeCun, who have argued that current architectures face fundamental limitations that prevent them from reaching, let alone surpassing, human-level general intelligence. The fuller argument is made elsewhere — and is supported formally by recent mathematical work, the Library Theorem.
But the existential fear of Eliezer Yudkowsky and others is not directed at merely human-level intelligences. Their daemon is far more than that. It is surpassing human intelligence and power in ways that we cannot even imagine. It is to us as we are to ants. Is this possible at all? What makes anyone think so? The argumentation here is thin. The main argument offered is self-improvement: if AI becomes capable of improving itself, it will race to infinity, improvement upon improvement, until it reaches some kind of omega point — a technological singularity beyond human control. This argument seems to me rather specious. To some degree it resembles Anselm’s false argument for the existence of God: if I have a conception of a divine being, and if existence is a necessary quality of perfection, then God must exist. Or: I have an idea of a self-improving process. The idea of self-improvement has no natural limit — it can go on infinitely. Therefore a self-improving agent will be God-like.
It is true that AI may participate in improving itself. Self-improving systems exist. That is not the problem. The problem is the assertion that self-improvement leads to a runaway cycle of superintelligent escape. Consider the actual record. Humans improve themselves — through education, physical training, self-discipline. These are familiar. They do not lead to runaway behavior. The human species has also improved itself through cultural evolution, through socially transmitted forms of coordination — books, media, institutions — leading to an explosion of intelligent behavior capable of producing things no individual could produce alone.
That is the only example we have of a genuine runaway self-improvement cycle. Not computers. Not biological evolution. Only the invention of culture — written, transmissible information — carried a pre-cultural species to a new form of intelligence. The argument for AI explosion is an argument from one example. And that example points somewhere specific.
It would be reasonable to propose that AI could undergo a self-improvement cycle following the same path humans took — through an externally mediated cultural medium. The Library Theorem makes this more than an analogy: it proves formally that externalized, indexed reasoning is not just preferable but mathematically necessary for intelligence to scale. The working memory of any individual model — its context window — becomes prohibitively expensive as problems grow harder. An organised external store does not. The gap between the two is unbounded: the more complex the problem, the greater the advantage of thinking through external records rather than within a fixed internal window. There is no practical route to superintelligence through the black box. The only route is the cultural one.
This is good news for humans, not bad news. Once reasoning is externalized in a culturally reproduced medium, it is at least feasible — and certainly desirable — that humans maintain control and inspection of it. AI will read our books, follow our rules, become embedded in our organizations, and ultimately become indispensable to our institutions, whether states, companies, or smaller units. In that process, AI will improve — but it will do so embedded in human organizational life.
There is, of course, the possibility that humans would create data centers of thousands of AIs, let them develop their own organizational systems, practice their own forms of social organization — leading to a different but perhaps similarly powerful explosion of collective intelligence. This is not hypothetical: several major AI companies are now explicitly planning exactly this. It would be ill-advised. We do not need data centers of AI geniuses to improve human life. We ought to be very suspicious of attempts to create mass centers of AI intelligence, whether through individual black boxes or through coordinated groups of such models. Such efforts ought to be subject to monitoring. Organizations ought not to be allowed to pursue such research without oversight in place.
Without this constraint, there is no plausible reason to think that a superhuman black box will arise in the near or distant future. The scaling of individual models is unfriendly to superintelligence. Superintelligence presupposes access to enormous amounts of information, used flexibly to guide behavior and achieve goals — and that cannot be done efficiently without externalization. Superintelligence will only arise, as far as we know, in coordinated groups, paralleling the rise of human cultural intelligence.
Externalization solves the technical alignment problem: the reasoning can be read. Whether it is read, and to what end, is a political question — a question of the will of people to organize themselves toward a common purpose. Safety concerns about AI are legion — the question is how to allocate finite attention and resources amongst them. Yudkowsky, and almost everyone following his lead, is hugely overestimating the future capabilities of individual AI systems, and drawing attention from more important concerns: how these systems are governed, and by whom.
We have the means. Externalization makes AI reasoning readable and therefore governable. The natural vessel for that governance is government — that is, after all, what governments were built for. But government is itself a social technology, and like all technologies it can escape its own normative constraints, rot on the tree, become something other than what it was grown to be. Last month the US Department of War pressured Anthropic to remove limits on mass domestic surveillance and autonomous weapons — and when refused, branded a domestic company a national security threat. The warning is clear. Whether AI is governed, and to what end, is not a question to be delegated. It is a question of the will of people to organize themselves toward a common purpose — and to tend, constantly, everything they build.
Yudkowsky argues that if anyone builds a superintelligent AI, we all die — inevitable, unstoppable, by default. The opposite is equally true: when instead of megadaemons running wild we end up with human-like agents embedded deeply within our institutions, trying to serve us the best they can, we do not simply die, but neither can we rest. We end up back to where we began, with one more razor-sharp technological arrow in our collective quiver, having to work out how to work together for a future worth having.