The discussion about existential risk from superintelligent AI is back, seemingly awakened by the recent dramatic progress in large language models such as GPT-4. The basic argument goes something like this: at some point, some AI system will be smarter than any human, and because it is smarter than its human creators it will be able to improve itself to be even smarter. It will then proceed to take over the world, and because it doesn't really care for us it might just exterminate all humans along the way. Oops.
Now I want you to consider the following proposal: Elden Ring, the video game, is an equally serious existential threat to humanity. Elden Ring is the best video game of 2022, according to me and many others. As such, millions of people have it installed on their computers or game consoles. It's a massive piece of software, around 50 gigabytes, and it's certainly complex enough that nobody understands entirely how it works. (Video games have become exponentially larger and more complex over time.) By default it has read and write access to your hard drive and can communicate with the internet; in fact, the game prominently features messages left between players and players "invading" each other. The game is chock-full of violence, and it seems to want to punish its players (it even makes us enjoy being punished by it). Some of the game's main themes are civilizational collapse and vengeful deities. Would it not be reasonable to be worried that this game would take over the world, maybe spreading from computer to computer and improving itself, and then killing all humans? Many of the game's characters would be perfectly happy to kill all humans, often for obscure reasons.
But if you believe in some version of the AI existential risk argument, why is your argument not then also ridiculous? Why can we laugh at the idea that Elden Ring will destroy us all, but should seriously consider that some other software - perhaps some distant relative of GPT-4, Stable Diffusion, or AlphaGo - might wipe us all out?
The intuitive response to this is that Elden Ring is "not AI". GPT-4, Stable Diffusion, and AlphaGo are all "AI". Therefore they are more dangerous. But "AI" is just the name for a field of researchers and the various algorithms they invent and papers and software they publish. We call the field AI because of a workshop in 1956, and because it's good PR. AI is not a thing, or a method, or even a unified body of knowledge. AI researchers that work on different methods or subfields might barely understand each other, making for awkward hallway conversations. If you want to be charitable, you could say that many - but not all - of the impressive AI systems in the last ten years are built around gradient descent. But gradient descent itself is just high-school mathematics that has been known for hundreds of years. The devil is really in the details here, and there are lots and lots of details. GPT-4, Stable Diffusion, and AlphaGo do not have much in common beyond the use of gradient descent. So saying that something is scary because it's "AI" says almost nothing.
(This is honestly a little bit hard to admit for AI researchers, because many of us entered the field because we wanted to create this mystical thing called artificial intelligence, but then we spend our careers largely hammering away at various details and niche applications. AI is a powerful motivating ideology. But I think it's time we confess to the mundane nature of what we actually do.)
Another potential response is that what we should be worried about systems that have goals, can modify themselves, and spread over the internet. But this is not true of any existing AI systems that I know of, at least not in any way that would not be true about Elden Ring. (Computer viruses can spread over the internet and modify themselves, but they have been around since the 1980s and nobody seems to worry very much about them.)
Here is where we must concede that we are not worried about any existing systems, but rather about future systems that are "intelligent" or even "generally intelligent". This would set them apart from Elden Ring, and arguably also from existing AI systems. A generally intelligent system could learn to improve itself, fool humans to let it out onto the internet, and then it would kill all humans because, well, that's the cool thing to do.
See what's happening here? We introduce the word "intelligence" and suddenly a whole lot of things follow.
But it's not clear that "intelligence" is a useful abstraction here. Ok, this an excessively diplomatic phrasing. What I meant to say is that intelligence is a weasel word that is interfering with our ability to reason about these matters. It seems to evoke a kind of mystic aura, where if someone/something is "intelligent" it is seen to have a whole lot of capabilities that we not have evidence for.
Intelligence can be usefully spoken about as something that pops up when we do a factor analysis of various cognitive tests, which we can measure with some reliability and which has correlations with e.g. performance at certain jobs and life expectancy (especially in the military). This is arguably (but weakly) related to how we use the same word to say things like "Alice is more intelligent than Bob" when we me mean that she says more clever things than he does. But outside a rather narrow human context, the word is ill-defined and ill-behaved.
This is perhaps seen most easily by comparing us humans with other denizens of our planet. We're smarter than the other animals, right? Turns out you can't even test this proposition in a fair and systematic view. It's true that we seem to be unmatched in our ability to express ourselves in compositional language. But certain corvids seem to outperform us in long-term location memory, chimps outperform us in some short-term memory tasks, many species outperform us for face recognition among their own species, and there are animals that outperform us for most sensory processing tasks that are not vision-based. And let's not even get started with comparing our motor skills with those of octopuses. The cognitive capacities of animals are best understood as scrappy adaptations for particular ecological niches, and the same goes for humans. There's no good reason to suppose that our intelligence should be overall superior or excessively general. Especially compared to other animals that live in a variety of environments, like rats or pigeons.
We can also try to imagine what intelligence significantly "higher" than a human would mean. Except... we can't, really. Think of the smartest human you know, and speed that person up so they think ten times faster, and give them ten times greater long-term memory. To the extent this thought experiment makes sense, we would have someone who would ace an IQ test and probably be a very good programmer. But it's not clear that there is anything qualitatively different there. Nothing that would permit this hypothetical person to e.g. take over the world and kill all humans. That's not how society works. (Think about the most powerful people on earth and whether they are also those that would score highest on an IQ test.)
It could also be pointed out that we already have computer software that outperforms us by far on various cognitive tasks, including calculating, counting, searching databases and various forms of text manipulation. In fact, we have had such software for many decades. That's why computers are so popular. Why do we not worry that calculating software will take over the world? In fact, back in 1950s, when computers were new, the ability to do basic symbol manipulation was called "intelligence" and people actually did worry that such machines might supersede humans. Turing himself was part of the debate, gently mocking those who believed that the computers would take over the world. These days, we've stopped worrying because we no longer think of simple calculation as "intelligence". Nobody worries that Excel will take over the world. Maybe because Excel actually has taken over the world by being installed on billions of computers, and that's fine with us.
Ergo, I believe that "intelligence" is a rather arbitrary collection of capabilities that has some predictive value for humans, but that the concept is largely meaningless outside of this very narrow context. Because of the inherent ambiguity of this concept, using it an argument is liable to derail that argument. Many of the arguments for why "AI" poses an existential risk are of the form: This system exhibits property A, and we think that property B might lead to danger for humanity; for brevity, we'll call both A and B "intelligence".
If we ban the concepts "intelligence" and "artificial intelligence" (and near-synonyms like "cognitive powers"), the doomer argument (some technical system will self-improve and kill us all) becomes much harder to state. Because then, you have to get concrete about what kind of system would have these marvelous abilities and where they would come from. Which systems can self-improve, how, and how much? What does improvement mean here? Which systems can trick humans do what they want, and how do they get there? Which systems even "want" anything at all? Which systems could take over the world, how do they get that knowledge, and how is our society constructed so as to be so easily destroyed? The onus is on the person proposing a doomer argument to actually spell this out, without resorting to treacherous conceptual shortcuts. Yes, this is hard work, but extraordinary claims require extraordinary evidence.
Once you start investigating which systems have a trace of these abilities, you may find them almost completely lacking in systems that are called "AI". You could rig an LLM to train on its own output and in some sense "self-improve", but it's very unclear how far this improvement would take it and if it helps the LLM get better at anything to worry about. Meanwhile, regular computer viruses have been able to randomize parts of themselves to avoid detection for a long time now. You could claim that AlphaGo in some sense has an objective, but it's objective is very constrained and far from the real world (to win at Go). Meanwhile, how about whatever giant scheduling system FedEx or UPS uses? And you could worry about Bing or ChatGPT occasionally suggesting violence, but what about Elden Ring, which is full of violence and talk of the end of the world?
I have yet to see a doomer/x-risk argument that is even remotely persuasive, as they all tend to dissolve once you remove the fuzzy and ambiguous abstractions (AI, intelligence, cognitive powers etc) that they rely on. I highly doubt such an argument can be made while referring only to concrete capabilities observed in actual software. One could perhaps make a logically coherent doomer argument by simply positing various properties of a hypothetical superintelligent entity. (This is similar to ontological arguments for the existence of god.) But this hypothetical entity would have nothing in common with software that actually exists and may not be realizable in the real world. It would be about equally far from existing "AI" as from Excel or Elden Ring.
This does not mean that we should not investigate the effects various new technologies have on society. LLMs like GPT-4 are quite amazing, and will likely affect most of us in many ways; maybe multimodal models will be at the core of complex software system in the future, adding layers of useful functionality to everything. It may also require us to find new societal and psychological mechanisms to deal with impersonated identities, insidious biases, and widespread machine bullshitting. These are important tasks and a crucial conversation to have, but the doomer discourse is unfortunately sucking much of the oxygen out of the room at the moment and risks tainting serious discussion about societal impact of this exciting new technology.
In the meantime, if you need some doom and gloom, I recommend playing Elden Ring. It really is an exceptional game. You'll get all the punishment you need and deserve as you die again and again at the hands/claws/tentacles of morbid monstrosities. The sense of apocalypse is ubiquitous, and the deranged utterances of seers, demigods, and cultists will satisfy your cravings for psychological darkness. By all means, allow yourself to sink into this comfortable and highly enjoyable nightmare for a while. Just remember that Morgott and Malenia will not kill you in real life. It is all a game, and you can turn it off when you want to.
