The other week, I was interviewed about the discourse around AGI and why people like Sam Altman say that we will reach AGI soon and continue towards superintelligence. I said that people should stop using the term AGI, because it's lazy and misleading. Here are the relevant paragraphs, for context:
Some people have asked what I mean by this. It would seem to be a weird thing to say for someone who recently wrote a (short) book with the title Artificial General Intelligence. But a central argument of my book is that AGI is undefinable and unlikely to ever be a useful concept. Let me explain.
What would AGI mean? An AI system that can do everything? But what is "everything"? If you interpret this as "solve every possible problem (within a fixed time frame)", that is impossible per the No Free Lunch theorem. Further, we don't even know what kind of space every possible problem would be defined in. Or whether such a space would be relevant to the kinds of problems humans care about, or the kind of thinking humans are good at. Comparing ourselves with other animals, and with computers, it seems that our particular cognitive capacities are a motley bunch occupying a rather limited part of possible cognition. We are good at some things, bad at others, even compared with a raven, a cuttlefish, or a Commodore 64. Psychologists claim that they have a measure of something they call "general intelligence", but that really only means factor analysis on a bunch of different tests they have invented, and different tests would yield a different measure.
But let's say we mean by AGI a computer system that is good at roughly the kind of thinking we are good at. Ok, so what counts as thinking here? Is falling in love thinking? What about tying your shoelaces? Making a carbonara? Understanding your childhood trauma? Composing a symphony, planning a vacation, proving the Riemann hypothesis? Being a good friend, and living a good life?
Additionally, there is the issue of whether these capabilities would come "out of the box", or do they need some kind of training, or prompting? How extensive would that preparation be? Humans train a long time to be good at things. How hard is it to instruct the AI system to use this capacity? How fast can it do it, and how much does it cost? How good does the end result need to be? Would an AGI system also need to be bad at things humans are bad at? And what about when it is unclear what good and bad means? For example, our aesthetic judgments partly depend on the limits of our sensory processing and pattern recognition.
One way of resolving these questions is to say that AGI would be an AI system that could do a large majority (say, 90%) of economically important tasks, excluding those that require direct manipulation of the physical world. Such a system should be able do these tasks with minimum instruction, perhaps a simple prompt or a single example, and it would do them fast enough (and cheaply enough in terms of computation) that it would be economically competitive with an average human professional. The quality of the end result would also be competitive with an average human professional.
The above paragraph is my best attempt at "steelmanning" the concept of AGI, in the sense that it is the most defensible definition I can think of that is relevant to actual human concerns. We can call it the "economic definition" of AGI. Note that it is much narrower than the naïve idea of AGI as being able to do literally anything. It excludes vast spaces of potential cognitive ability, including tasks that require physical manipulation, things we haven't figured out how to monetize, things that cannot easily be defined as tasks, and of course all kinds of cognition humans can't carry out or have not figured out how to do well yet. (We are very bad at coming up with examples of cognitive tasks that neither we or our machines can do, because we have constructed our world so that it mostly poses us cognitive challenges we can handle. We can call this process civilization.)
Alas, even the economic definition is irredeemably broken. This is because which tasks are economically important is relative to our current economy and technology. Spellchecking is not a viable job for humans because computers do that now; typesetting has not been a viable job since desktop publishing; and once upon a time, before the printing press, manually copying texts ("manuscripts") was an intellectual job performed by highly trained monks. Throughout human history, new technologies (machines, procedures, and new forms of organization) have helped us do the tasks that are important to us faster, better, and simpler. Again and again. So if you take the economic definition of AGI literally, we have reached AGI several times in the history of civilization.
Still, unemployment has been more or less constant for as long as we have been able to estimate it (when smoothed over a few decades). This is because we find new things to do. New needs to fulfil. As Buddha taught, human craving is insatiable. We don't know in advance which the new jobs will be and which kind of cognitive skills they would require. Historically, our track record in predicting what people will work with in the future is pretty bad; it seems that we are mostly unable to imagine jobs that don't exist yet. There have been many predictions that we will only work a few hours a day, or even a few hours per week by now. But somehow, there are still needs that are unfulfilled, so we invent more work. Most people today work in jobs that would be unimaginable to someone living 200 years ago. Even compared to when I was born 45 years ago, people may have the same job titles (graphic designer, travel agent, bank teller, car mechanic etc) but the actual tasks done within these jobs are quite different.
One attempt to salvage the economic definition of AGI would be to say that AGI is a system that can perform 90% of the tasks that are economically valuable right now, January 2025. Then AGI will mean something else next year. This sounds like a viable definition of something, but I would have expected this much talked-about concept to be a little less ephemeral.
Alternatively, you could argue that AGI means a system that could do 90% of all economically valuable tasks now, and also all those that become important after this system is introduced, in perpetuity. This means that whenever we come up with a new need, an existing AGI system will be ready to satisfy that. The problem with this is that we don't know which tasks will be economically important in the future, we only know that they will be tasks that become important because AGI (or, more generally, technology) can do the tasks that were economically important previously. So… that means that AGI would be a system that could do absolutely everything that a human could potentially do (to some extent and capacity)? But we don't even know what humans can do, because we keep inventing new tasks and exploring new capacities as we go along. Jesus might have been a capable carpenter but could neither know that we would one day need software engineering nor that humans could actually do it. And we certainly don't know what humans will find important in the future. This definition becomes weirdly expansive and, crucially, untestable. We could basically never know whether we had achieved AGI, because we would have to wait for decades of social progress to see whether the system was good enough.
This is getting exhausting, don't you think? This initially intuitive concept got surprisingly slippery. But wait, there's more. There are a bunch of other definitions of AGI out there which are not formulated in terms of the ability of some systems to perform tasks or solve problems. For example, pioneering physicist David Deutsch thinks that AGI is qualitatively different from today's AI methods, and that true AGI is computationally universal, can create explanatory knowledge, and can be disobedient. Other definitions emphasize autonomy, embodiedness, or even consciousness. Yet other definitions emphasize the internal working of the system, and tend to exclude pure autoregressive modeling. Many of these definitions are not easily operationalizable. Most importantly, they are surprisingly different from each other.
Now, we might accept that we cannot precisely define AGI, and still think that it's a useful term. After all, we need some way of talking about the increasingly powerful abilities of modern AI, and AGI is as good a term as any, right?
Wrong. It's lazy and misleading. Why?
Lazy: Using the term AGI is a cop out of having to be clear about which particular system capabilities you are talking about, and which domains they have impact on. Genuine and impactful discussion about the progress of AI capabilities and their impacts on the world requires being concrete about the capabilities in question and the aspects of the world they would impact. This requires engaging deeply with these topics, which is hard work.
Misleading: As the term AGI will inevitably mean different things for different people, there will be misunderstandings. When someone says that AGI will arrive by time T and it will lead to X, some people will understand AGI as referring to autonomous robots, others as a being with godlike powers, yet others as digital copy of a human being, while the person who said it might really just mean a souped-up LLM that can write really good Python code and convincing essays. And vice versa. None of these understandings is necessarily wrong, as there is no good definition of AGI and many bad ones.
Misleading: The way the term of AGI is used implies that it is a single thing, and reaching AGI is a discrete event. It can also imply that general intelligence is a single quantity. When people hear talk about AGI appearing at a certain date, they tend to think of time as divided into before and after AGI, with different rules applying. All of those are positions you can hold, but which do not have particularly strong evidence in their favor. If you want to argue those positions, you should argue them separately, not smuggle them in via terminology.
Misleading: To many, AGI sounds like something that would replace them. That's scary. If you want to engage people in honest and productive discussion, you don't want to start by essentially threatening them. Given that the capabilities of existing, historical, or foreseeable AI methods and systems are very uneven (what Ethan Mollick calls the "jagged frontier") it makes most sense to talk about the particular concrete capabilities that we can foresee such systems having.
I would like to clarify what I am not saying here. I am not saying we should stop talking about the progress of AI capabilities and how they might transform society. On the contrary, we should talk more about this. AI capabilities of various kinds are advancing rapidly and we are not talking enough about how it will affect us all. But we need to improve the quality of the discussion. Using hopelessly vague and ambiguous terms like AGI as a load-bearing part of an argument makes for bad discussion, limited understanding, and ultimately bad policy. Everytime you use the term AGI in your argument you owe it to yourself, and your readers/listeners, to replace it with a more precise term. This will likely require hard thinking and might change your argument, often by narrowing it.
I would also like to clarify that I am accusing a whole lot of people, including some rich and/or famous people, of being intellectually lazy and making misleading arguments. They can do better. We can all do better. We should.
Not everyone argues this way. There are plenty of thoughtful thinkers who bother to be precise. Even leaders of large industrial AI labs. For example, Dario Amodei of Anthropic wrote a great essay on what "powerful AI" might mean for the world; he avoids the term AGI (presumably because of the conceptual baggage discussed here) and goes into commendable detail on particular fields of human enterprise. He is also honest about which domains he does not know much about. Another example is Shane Legg of DeepMind, the originator of the term AGI, who co-wrote a paper breaking down the concept along the axes of performance and generality. It is worth noting that even the person who came up with the term (and may have thought deeper about it that anyone else) happily acknowledges that it is very hard to define, and is perhaps better seen as a spectrum or an aspiration. The difference between us is that I think that such an acknowledgement is a good reason to stop using the term.
If you have read all the way here but for some reason would like to read more of my thoughts about AGI, I recommend that you read my book. It's short and non-technical, so you can give it to your friends or parents when you're done.
If you find yourself utterly unconvinced by my argument, you may want to know that I gave this text to both Gemini, Claude, and R1, and they thought it was well-argued and had no significant criticisms. But what do they know, it's not like they are general intelligences, are they?
No comments:
Post a Comment