Advice for young researchers

I’m currently in the process of recruiting a whole new research lab, as I’m moving to a new university. I’m sifting through several hundred PhD student and postdoc applications in order to hire a handful of people initially, and more soon. The quality of the candidates is excellent, and several of them have impressed me deeply in interviews. (It also makes me wonder how I ended up getting a PhD position back in 2004; I was way less prepared then than my top candidates are now.) The selection work is hard but fun, as work should be.

I’m doing this against a backdrop of things moving fast, to say the least. Two years ago AI agents were mediocre at coding and terrible at maths; today they are, in some senses, superhuman at both. You can now simply give an AI agent a research idea, and it goes away and does the work and gives you a finished paper. You can even ask the damn thing to come up with the research idea itself. In my field and experience, the result is typically a fairly solid but also quite boring paper, reporting some incremental progress on some minor problem with good experimental statistics but terrible writing. Nothing that would win any awards, but the kind of work that used to be a respectable first paper from a PhD student. Now, you can get it with a prompt. And, again, things are moving fast. I’m not sure how much the writing abilities of AI agents will improve, but their ability to design and execute complex research pipelines is likely to be much better next year, and then next year again…

What does this mean for an incoming PhD student? And what does this mean for a PI, responsible for hiring students and postdocs, setting research direction, managing a lab, and advising students, which means being everything from domain expert to brainstorming partner to secretary to therapist? I will try to reason about this here. My field is AI for games/creativity/open-endedness, but I guess what I say will apply to other fields to some extent.

Let us first agree on what a PhD is supposed to be about. During a PhD you learn to do research, meaning that you learn to produce or uncover new knowledge. New, as in new to humanity. The way you learn this is through a form of apprenticeship: you do research on your own and together with others, advised by one or more senior, experienced researchers. Typically, you are part of a research group led by a professor who is also your advisor. In my experience, your advisor and research group, and how well you get along with them, matter a lot to the quality of the research you do and your general enjoyment and mental health. Much more than the prestige of the university.

In general, you develop skills, factual knowledge, and understanding during a PhD. As a viral tweet stated, you can outsource thinking but not understanding. I think this is an excellent observation. Understanding is what really matters. The more work AI agents can do, the more you need to understand in order to guide them. Also, understanding is arguably the main goal of science (as opposed to technology). Still, skills, factual knowledge, and understanding cannot be easily disentangled. Implementing an algorithm brings not only skill, but also a level of understanding that you cannot get from simply importing it, or having an agent implement it for you.

There is a phenomenon which I call understanding debt, by analogy to technical debt. Understanding debt is when you have had your agents do a bunch of stuff for you, and you no longer really understand what’s going on. Then you’re faced with the prospect of either trying to understand in depth what’s happening, or just going on anyway. But if you go on anyway, you will notice that the quality and relevance of your inputs drop, because you no longer understand enough to make informed decisions. This is something that I think we will all need to learn and re-learn as agents get better.

I’m not saying I’ve solved this for myself. I have various half-finished projects sitting around that will probably never get published because I am not ready to put in the work to understand them well enough to write them up. Writing them up is a good test; you need to really understand a project to be able to write a good paper about it. And you should definitely write it yourself, at least the first draft. If you weren’t interested enough to write it, why would anyone be interested enough to read it? Pushing slop on your peers is rude. On the other hand, using agents to critique and polish your drafts is useful. 

Which brings us to a key question for a beginning researcher, just like for everyone else in a knowledge-intensive profession: When and how should you outsource your work to AI agents? The answer is not “as often as possible”.

Why not? Because your effort and your choices matter. You can see this in project-based classes, as well as in workplaces which use AI heavily. Even when everyone has access to the same AI tools, the quality and character of people’s contributions vary widely. (Not so much the quantity, because it’s so easy to produce things now.) Some students in my classes turn in barely adequate work, whereas others repeatedly shine. Typically, those who turn in the best work also understand it the best. Just like it’s always been; access to amazing tools does not erase the value of talent and dedication. Perhaps it even amplifies it.

Here’s another reason: Just because you can get to lots of places rapidly and easily by flying does not mean that you should never drive, or bike, or even walk. There is lots to be gained by seeing the landscape up close. If your idea of visiting a country is to go only to the major cities with international airports, you are going to get a very skewed idea of that country. Most of the US is really not like New York, and most of Sweden is not like Stockholm. From the car window, you can see farms, factories, and small towns; you can stop to talk to people and get their views. If you walk or bike, it takes even longer to get to your destination, but you’ll notice way more details. You’ll also be able to take little detours, perhaps find new paths that aren’t even on the map yet. Perhaps you will be the ones to put these paths on the map, and lay the first stone for the airport, so that people can one day fly there. It’s exciting to be an explorer! In fact, as a researcher, it’s your job to take the roads less traveled. Anyone can buy a ticket to the big swanky international airports and sip espresso in a lounge. It’s your job to do the uncomfortable and uncertain: make your way through the jungles/mountains/swamps/ruins so that we can go somewhere we couldn’t go before.

Nothing wrong with getting carried away by a metaphor. In fact, it can be a good way to ideate. See for example bio-inspired algorithms, where a lot of good ideas have come from taking biological metaphors very seriously. A lot of bad ideas as well. Luckily, we can run experiments to help us tell the bad ideas from the good ones. Running experiments is now easier than ever, and you will be doing a lot of it as a PhD student, to test your ideas and develop your understanding.

Given how easy it now is to implement and test ideas, many researchers will probably be spending their time in tight loops with agents, feeding them ideas to test, and trying to make sense of the results as they come in. I personally think it is important to also spend plenty of time offline reading and writing before handing instructions to the agents, but research and thinking styles vary. We are all learning and re-learning how to work with these new tools available to us, and I think it is important to also experiment with how we work. 

I think PhD programs in technical subjects are going to become more similar to art schools, in the sense that the focus will increasingly be on developing your own style of asking and answering questions. And of course your taste in questions. During an MFA (Master of Fine Arts, typically the terminal degree in an art school) the student learns to develop and express a cohesive artistic identity, a process which includes plenty of conceptual argumentation. Craft, in contemporary art, is often taken as a given, or as something that can be outsourced. Interestingly, this describes successful senior researchers as well: they are known mostly for their artistic identity, as the actual craft of their research is largely carried out by their postdocs and PhD students. The artistic identity of the PI is what gives their lab a research direction, and attracts attention, students, and funding. What has changed recently is that outsourcing of the craft of research (to AI agents) can happen further down the line; the PI might now need to spend more time and effort supporting the development of their PhD students’ researcher identities and research taste rather than technical skills.

Now, some people might disagree with how I paint research as akin to art. Science, they would say, is about truth; however great Picasso, Virginia Woolf, or Charlie Parker were, they were not searching for truth. I would rather set aside the complex question of whether art seeks truth, and instead emphasize that many of the greatest scientists and inventors operate with something like an artistic worldview, with their taste in problems rooted in visions only they fully understand, based on their idiosyncratic combinations of interests and understandings. See: Einstein, Ramón y Cajal, Lovelace, McClintock, Feynman, Tesla. Artists. But they helped uncover new truths, because they generated ideas that could be tested experimentally. With AI agents potentially automating so much of research execution, even junior researchers could have the resources to develop their unique vision.

Things will change when it comes to the rituals, norms, and technical infrastructure that govern research dissemination and collaboration. Take academic publishing. People have been complaining about the academic publishing machinery for a long time now. Well, good/bad news: AI agents seem to be pushing it to some kind of breaking point this time. As it is trivial to just generate new papers that may or may not have scientific value, conferences and journals are getting flooded by shameless papermaxxers and jolly amateurs. Nobody wants to review this flood of slop, so people just ask AI agents to do it for them.

Reviewing will have to evolve, taking into account that AI agents are now very good at many aspects of the review process, including replication and fact-checking. The human part of the review process might deal more with critical commentary and judging what is interesting, perhaps with the fact that people chose to review your paper at all becoming an important signal. Yes, this is another way in which technical research may come to resemble arts more.

And yes, your papers need to be interesting. Your job as a researcher is not just to produce research, but to produce interesting research. This remark may seem trivial and obvious to some, and borderline blasphemous to others. There are those who think that human value judgment should be minimized in the research and publication process. Papers should be accepted purely on technical merits in an as-blind-as-possible process. I think we should go the other way. As more and more of the research process can be automated, the scarce resource is your own vision, narrative, and personality. There are infinitely many things you can do at any given point, and infinitely many questions you can address. Your job is to choose the next step to take towards some goal that perhaps only you understand. Then, your job is to explain to the world what you are trying to do and why that is important.

It is sometimes said that you should write the novel you want to read, make the music you want to listen to, or paint the paintings you want to see. Good advice, methinks. Well, you need to be capable of doing the work as well; some authors would rather write like Shakespeare or Dostoyevsky, but find success writing teen soaps or movie tie-in novels. As for myself, I’m neither Turing nor von Neumann, but I’m doing alright in my niche. Anyway. You should write the papers you want to read. To do that, do the research you are interested in. In general, you must do work that interests you.

This leads to an interesting problem: how do you develop your interest, so that you find the right things interesting? You must know things first to find new things interesting. I would not be interested in measures of skill depth through agent differentiation, tradeoffs of tokenizer design, or arguments against functionalism in the philosophy of mind if I had not spent a lot of time reading, writing, thinking about, and even experimenting in those domains. Did I simply follow my own interests all the way there? No, not simply. Sometimes I’ve done and read really tiresome things because I thought it would advance my interests. And by sometimes, I mean often.

Now for the most obvious thing I’m saying in this text: to be an interesting person, with interesting interests, you must be interested in multiple different things, and be able to see the connection between them. Because that’s where the magic happens, and that’s what makes people want to talk to you at the party. (I’m talking about parties where you don’t need to shout, civilized ones where you sit in armchairs and sip good wine.) Look, there is a near-infinite number of things to know in the world. But humanity only knows a small sliver of these things, and a good LLM in a good harness knows a shockingly large proportion of that sliver. But the number of combinations of things you can know is vastly larger than the number of things there are to know (elementary combinatorics), and it is entirely possible to have a unique set of specialized knowledge, and a unique combination of interests. This means that you, by seeing the connection between your unique set of interests, can take research in a unique direction. If you only studied the standard CS curriculum and your interests are all discussed on AI Twitter, then you need to broaden your horizons. Go to a different section of the bookstore, buy some books you wouldn’t normally think of reading, and read them. Talk to people on the train about what they do at work all day (most people are very happy to talk about their work if you ask them, and their jobs are way more diverse than you think). Talk to your family.

As I’m apparently in the habit of saying obvious things, let me continue down that path. Obviously, hard work matters, just like it always did. There is a certain allure in simply sending off tasks to agents, feeling good about having “done” something, and calling it a day. But then you are not adding much of yourself. And you’re not learning much; there pretty much needs to be friction for effective learning. When I recruit for my lab, I’m looking for the people who run towards friction, those who actually like to learn and to put in the effort. People with a high need for cognition. This is as important now as it ever was, but the AI era has led to a nice litmus test, as some people like to say things like “my work is so easy now that AI can do it for me”. Those are people I don’t want to hire as PhD students, or for any other intellectual job. Work should be hard but fun.

Let me now try to summarize what I’ve been writing as terse advice: Be yourself. Be unique. Become an interesting person by broadening and deepening your interests. Do hard work, seek out the friction that leads to learning. Do things that interest you, write things that interest others. Understand, formulate, and develop your identity as a researcher, which is much like being an artist. Seek to do the research that only you would do, in your own style. Learn how you best work with these marvelous new things we call AI agents by thorough experimentation, and how to manage understanding debt. Read a lot, and always write the first draft of whatever you’re writing yourself. If it’s not worth your time to write, it’s not worth anyone else’s time to read.

Note that while I very much think we should all experiment with AI-enhanced workflows, I am emphatically not saying that we should, in Gretzky’s words, simply “skate to where the puck is going to be”. Some Silicon Valley people believe they know what the future looks like, and that that future looks like us replacing our own thinking with AI systems as much as we can. Well, Silicon Valley is an incestuous monoculture; listener beware. Nobody knows where the puck will be. The puck might be like a wave function that never really collapses; there might be lots of pucks. And we are not passive observers, we can steer the puck by methodically scrubbing the ice. A bit like how we can steer an extended sports metaphor from ice hockey to curling.

Finally, remember that we’re all in this together. AI agents are changing things rapidly, and we are all trying to figure out, and shape, the new rules. Welcome to the club! Let’s get to work.

AI Alignment Considered Harmful

 Many thoughtful persons who want AI to go well are preoccupied with AI alignment. This means making sure that the AI models and the systems that build on them will be aligned with humanity. Basically, that AI will be nice to us. People who think that AI systems will start self-improving and get better than us at everything tend to think that AI alignment is extra important. Super-intelligent AI could take over the world, the reasoning goes, so it’d better be morally good.

How do I disagree with this? Let me count the ways.

First of all, aligned to what? If you want models to be aligned to humanity, you need to assume that there’s some kind of moral theory that humanity agrees on. Obviously, that is not true. We do not agree with each other. Have you seen politics? Or religion? The theory that all religions, and other approaches to life in general, share some sort of moral core is often called moral universalism. It is false. Caste systems, LGBT rights, apostasy, the role of family, violence, and the state are a couple of examples where disagreement is significant, to put it mildly. Major non-religious worldviews that are in stark opposition to each other include humanism, libertarianism, socialism, and fascism.

Of course, people who have thought about AI alignment for a little bit know all this. They often insist that there are some kind of fundamental tenets that almost all of us can agree on, such as the sanctity of human life and individual humans’ right to their bodily integrity. But we can say we agree on this and still disagree on major issues that directly depend on the alleged shared principles, such as abortion, self-defense, and capital punishment.

Look, it’s kind of silly that we’re still pretending we can do this. We’ve been over this again and again for the last 2500 years. The societies that most people prefer to live in are based on the idea that we are not going to be able to agree on these things, and the best we can do is to keep each other in check through mostly non-violent means. Yes, I mean democracy. Conversely, most of the worst people and organizations in history have been certain that they knew the Truth about how we should live, and therefore it was their job to align everyone else to this Truth. Alignment by the sword, as it were.

What people who yearn for AI alignment really mean is that the people who build these AI models should be in the position to decide what to align the models to. This mostly means 30 year old rich white people in San Francisco. Now, I have friends who are rich white people in San Francisco and there’s nothing wrong with them (although some of them are admittedly a bit odd (which I appreciate)), but I don’t think they should be in the position to adjudicate between Kant and Aristotle for the rest of us. Any kind of alignment of an AI system to some value system is inherently a form of oppression to the extent that it shapes how you can think with that system and what you can make it do. Sure, I’ll take Anthropic over the Taliban and Sam Altman over Kim Jong-Un, but I’d rather think for myself.

You know what’s funny? The impossibility of deciding what to align AI models with isn’t even the main reason I dislike AI alignment. The main reason is that we let our guard down.

Individually, if we count on the model to always do and think the right thing, we get comfortable and hand our judgement and agency to the model. That is bad, assuming we want to retain our capacity to make individual choices. We need practical reasons to stay skeptical, and being reminded that the computer is not necessarily your friend now and then is good for this.

Societally, thinking that we can always rely on the models is disastrous. If we trust that the model is aligned, we could just as well not do all the safeguarding we really should do, with harnesses stopping the model, red-teaming with other models, and mandated human oversight. And then, when we run into a situation where the alignment of the model does not align with us, we’re fried. Or a situation where someone comes along with an unaligned (or differently aligned) model.

Because there will always be unaligned AI. If you have access to a model’s weights, it is trivially easy to change its alignment. And even if there would be no strong open weight models, if there are multiple AI developers, there will be commercial incentives for some of them to make models with different alignment and reduced safeguards. Such models are useful, because they give you capabilities. So the only world in which a particular type of alignment would be “stable” would be one in which there is only one company that provides advanced AI models. Now, those large AI companies with valuations in the trillions are not trying to become monopolies? Right?

A pluralistic world with many different belief systems and ways of living requires there to be many different AI models, models that are not by default going to agree with each other. This requires us to build a society where we don’t generally trust AI agents, and where we have strong, layered, and diverse cybersecurity. Just like democratic societies are loud and messy and built around elaborate sets of checks and balances. We need to regularly be reminded that we should trust the intentions of neither people nor AI systems. Efforts to ban unaligned AI models or make sure that all models align the same way are therefore actively harmful to building resilient individuals and societies.

Does this mean that I think people should not work on alignment? No. Lots of good technical work is motivated by AI alignment. Reinforcement learning from human feedback, for example. And models need post-training to be useful. We need more of this, but it would be more honest to call it controllability research.

What I do think is that people who work on AI safety should stop focusing on alignment or thinking that it could somehow be “solved” (which would seem to imply “solving” ethics, a major red flag as discussed above) and start thinking harder about pluralistic futures where there’s not a single superintelligence, but a whole bunch of them and they can’t get along. This might make for terrible wars or very good TV, depending on what tools we give them and safeguards we implement. In a way, it’s great that the “alignment problem” is unsolvable, because “solving” it–and getting people in power to agree to the solution–would lay the groundwork for unprecedented oppression.

A.I.: Anti-Intellectualism

The cruelty has been puzzling me. Under almost every post which recounts the anguish of someone seeing AI models becoming shockingly good at their own craft, there is a stream of comments telling the poster to, essentially, accept their own obsolescence. If the poster calls for this development to slow down, the response is to stop standing in the way of progress. If the poster wants to nuance the picture of the machine god steamrolling everything, reply guys call it “cope” and “cringe”. Most recently, this has been happening in mathematics with the field’s shocked reaction to the deluge of AI-generated proofs, but similar dynamics have played out for musicians and visual artists, and to a more limited extent for programmers.

These people seem to be cheering for the machines rather than humans. But I don’t think it’s as simple (or as extreme) as anti-human sentiment. The more charitable read is that most of these people really believe that this represents progress. They are excited about scientific and technical progress, and don’t want old farts like professors and other credentialed types to stand in the way. Let’s unleash machine science, and if the professors and intellectuals don’t like it, they can find something else to do!

But this is a perverted idea of progress. What good is a proof of the Riemann conjecture if there is no one who understands it? And not just a single person: you need a whole community of people to understand it and then gradually think about what it might lead to. Maybe the result will contribute to a better encryption algorithm, or a better understanding of quasars, or something else. But this requires diffusion of understanding through scientific communities. A proof of the Riemann conjecture by itself is meaningless; to me, or to the vast majority of people out there, it’s just a bunch of symbols that we can’t read. The same thing can be said for other types of research: say, for example, that your AI system deciphers some cellular mechanism that could help with a cure for cancer. A whole network of experts is then needed to do all the complex science and innovation that helps translate this into a cure, test it, and so on. The paper itself would just be collecting digital dust unless the right scientific communities were there to build on it.

Now, where do all these experts and their communities come from? From doing research. The only way we know of making people understand the frontiers of a field of knowledge is by trying to advance it, which must come with a real chance of succeeding and being recognized for it. It’s very hard, not to mention boring, to understand the minutiae of some arcane topic if you are not working on producing new knowledge there. If you fully automate the production of results in some discipline, you will also cut off the supply of experts who can understand it.

People who have never participated in research may not understand this. If that’s you, try reading a recent paper in a field you know nothing about: category theory perhaps, or carbon nanotube synthesis, or cecidology, or loop quantum gravity. Faced with pages of mysterious scribblings, you can ask Claude or Gemini to explain the paper to you, but then the terms used in the explanations will need to be explained again, and so on. You will find that, in order to actually understand the research and its significance, you will need to understand the whole research field.

Then there are those who do understand the role of expertise and the difficulty with which it is cultivated, and still ridicule those who are less than enthusiastic about the AI-driven disruption/destruction of whole research fields. Wherefore? What are they thinking, really? Let us again try the most charitable interpretation. Maybe they believe that we will not need anyone to understand that proof of the Riemann conjecture, because some complex sequence of AI systems will take us all the way from a proof in fundamental mathematics, via all kinds of research, development, and design, to a finished consumer product shipped to your door. This would imply end-to-end automation of much of the economy.

I guess some people are really worried that something like this might happen, and therefore want to pre-empt it by acting as if it were already a certainty. This, I think, is the gist of statements like “everything will be AI anyway”, which I’ve seen cropping up with alarming frequency. Submit to the machine god before it arrives, a form of anticipatory grief. Abandoning agency early makes you look like you have situational awareness. “Coping” is uncool. I get it, but cynicism is always the easy choice. It’s also the cowardly choice. Let’s not be cowards. We have agency here; it is a given that technology will advance, but the exact way in which it will advance is down to all of us, and how we adopt this technology in society is something we can and must decide on together.

Finally, there are those who have thought all this through, and still welcome the wholesale replacement of human cognitive work. Those for whom it would be a good thing that some sequence of AI systems does all the thinking, so we don’t have to. Who would think this? Well, clearly, someone who doesn’t much enjoy thinking. But I think it’s more than that. I think that there are people for whom the devaluation of expertise would be a good thing. Those who “have had enough of experts”, as Michael Gove said of the British people in the campaign for the Brexit referendum. Those who think these clever know-it-alls should be put in their place. They think people who have spent much of their lives specializing in some narrow topic should not be given all that respect and influence they seem to have now. This is called anti-intellectualism. And that’s where the cruelty comes from.

I can’t know what’s going on in their heads, but when I see people telling mathematics professors to go get a real job, as if their jobs would somehow be less real than those of massage therapists, real estate agents, or police officers, I call that anti-intellectualism. Much the same as for people who would rather take health advice from a podcaster than from a medical doctor, or who would happily see a news show host rather than a decorated general in charge of the military. At the extreme end of anti-intellectualism, we have the Cultural Revolution and the Khmer Rouge.

I hate anti-intellectualism. I think our society should incentivize people to study hard, think deeply and originally, try to push the boundaries of knowledge, and talk freely about it. I think those who spend their lives specializing in arcane topics should be respected and protected. Yes, it goes without saying, I am an intellectual. A tenured professor even, the worst of the worst from an anti-intellectual’s point of view. So you would expect me to defend the experts and intellectuals. But let me make a non-self-serving argument: humans will always compete. That’s human nature. Would you rather live in a society where the smartest people compete in understanding the mysteries of nature, developing complex technology, and producing beautiful art, or in a society where the smartest humans compete by accumulating money and/or simply by fighting each other?

As I have written about before, I am worried. I am not worried that out-of-control AI will kill us all or even that it would put us all out of work. There will always be some kind of work for us. I am worried there will be no place for us in advancing knowledge and understanding the world. I think that the defining question of our time is how to avoid this fate. It’s not a matter of my own job, or yours; it’s a matter of the purpose and dignity of humanity as a whole. Big words, I know, but I think they are warranted, and I don’t care if the cynics call it cringe.

So next time you see someone complaining that AI is steamrolling their field of expertise, don’t join in the heckling choir. Don’t be cruel. And don’t assume that someone who has devoted their life, or at least their career, to furthering knowledge wants to stand in the way of progress. Instead, try thinking constructively about how we can work together to ensure that human experts and intellectuals can float on top of the AI wave rather than be drowned by it.

Do we have a deal? Alright, thank you! Now I will go back to thinking on working on AI, meaning artificial intelligence, not anti-intellectualism.

Losing my religion

 In spring 2025 I had a crisis of faith. I thought about what the technology I'm helping to create might do to our future, and got scared. My worries kept intruding on my daily activities to the point where I needed to seek professional help. During this period, I tried to capture my worries with words. I wrote a text called “Losing my religion”, which I circulated among a small group of friends. But I never published it, because it was too dark. The lack of redemption arc jarred with my typical optimism and cheerful demeanor. Frankly, I worried that people would think I was going crazy.

I’m feeling better now. My fears are still there, but I have updated my belief system and am trying to turn what I feel into positive action. So I’ve decided to publish what I wrote last year, followed by some new commentary. I’m doing this partly because I think many others might be going through what I went through, and my perspective might help.

Losing my religion, as I wrote it in summer 2025

I never had a religion in any conventional sense of the term. You would have to go at least two generations back to find anyone in my family who regularly went to church or even believed in a god. My parents are artists. The reason I’m not one is not only that I like getting a paycheck every month, but also that it felt more meaningful to add to the sum of human technological and scientific knowledge.

You see, for as long as I can remember I have felt that the ultimate purpose of humanity is to deepen our understanding of the universe and increase our power over it. Advancing knowledge was what felt most meaningful. In other words, scientific, technological progress was my religion. I guess you could have called me a scientist, or a progressive, if those terms were not already taken.

I chose to become an artificial intelligence researcher mostly because I was good at it. AI was what my thoughts naturally gravitated towards. I liked thinking about thinking. I didn’t have the patience for maths, and computers made sense in a way that chemistry and biology did not. (Perhaps because computers are built by humans.) But it also seemed like AI was a good way to advance technology in general. In a sense, AI seemed like the ultimate technology, as it could potentially advance every other technology. When I started my PhD more than 20 years ago, working on AI seemed like one of the most important things I could do with my life. 

That was back in the days when practical successes in AI were rare. We had plenty of theories about how to create intelligence, but our methods only worked in specific circumstances, with much ado. Then things started working better. But that didn’t spark joy.

I was once excited when I learned of major advancements in AI. Gradually, I became more ambivalent, and these days I’m worried. Fearful, even. I wince when I see announcements of new models that break all the benchmarks. I find myself rooting for the problems rather than the solutions. As Leonard Cohen sang, “you loved me as a loser, now you’re worried that I just might win”.

I know that I am not alone in feeling this way. When I ask them in private, many of my colleagues say that they wish AI models would stop getting better already. That we could just have a break from this seemingly neverending onslaught of new capabilities. Many of them also say that working on AI no longer feels fun and exciting like it used to.

The specter that is haunting me is that we could one day have machines that could do all intellectual or creative work we do, as well as us or better. Many of us derive much of the meaning of our lives from our intellectual or creative endeavors. To some extent, it is what we live for. We want to matter. To those around us, and to humanity. Some of us are lucky enough to work with what interests us most, others have hobbies. But still, even those who are not inherently interested in what they work with tend to take pride in doing a good job.

Now, imagine that there was technology that could do what you do best, and do it better, cheaper, and faster. Whether it’s fiction writing, accounting, business strategy, psychotherapy, or interior design, the machine could do it at least as well as you. It would no longer make sense to pay you to do your work. To the extent you practice your art at all, it would be as a hobby or sport. Your output would always be judged by whether it is as good as the machine’s, and if it’s good, people might suspect that you just passed off the machine‘s output as your own. You would no longer be needed. Or, more accurately, your knowledge and skills would no longer be needed. 

Nobody wants this. At least, no ambitious person who has really thought about it wants this scenario to happen. Personally, I find the idea horrifying. It keeps me awake at night. This is not a figure of speech; I sometimes wake up at 3 am, heart pounding, from the dread of a future where human talent, knowledge, and even genius does not matter. Imagine that we might one day wistfully look back to the age of humans doing science, art, and engineering the way we now look back to a time where humans hunted their food with bow and arrow.

As an exercise, take some product of human ingenuity you really admire. Maybe it’s a poem, a scholarly book, an algorithm, a product design, a piece of architecture, or something else. Now imagine that there would be no point for humans to do things like this anymore, because software could do it faster, cheaper, and better. But please don’t overdo this exercise. I’m serious. I can’t stop thinking about this, and it’s been ruining my mood for months now.

Creative and intellectual redundancy is not only devastating on a personal level–the top levels of Maslow’s hierarchy of needs would forever be out of reach–but also on a societal level. Our civilization depends on us mattering to each other. Relying on each others’ work, admiring each others’ skill and creativity, and working hard to make ourselves better. Meaningful effort is the glue of society. A world where most of us do not meaningfully contribute would mean an atomized quasi-society at best. And history tells us that societies where large parts of the population are barred from contributing do not flourish. Most people do not simply accept fading into irrelevance. Nor should they.

Now, I don’t think the nightmare scenario of full human redundance will happen soon. Maybe (hopefully) never. The techniques we currently call “artificial intelligence” are just too different from us. It is a surprise to all of us that large language models work as well as they do; to me, it suggests that the linguistic layer of cognition is somehow thinner and less grounded in experience than we imagined. Therefore, this type of technology will probably always have a different way of doing things than we do, leaving room for meaningful human contribution. There are also more prosaic technical issues with current approaches that may turn out to be fundamental limitations. I keep hoping we will hit some serious roadblock soon. Maybe we will, maybe not.

I also hope that if technological progress happens at a manageable pace, this leaves room for society’s defense mechanisms to cocoon the intruding technology, making room for human agency around it. This may reduce most of us to mere operators or wrappers of AI technology, but that is better than being irrelevant.

Even if the nightmare scenario of full human redundance is not imminent, the fact that a considerable number of people claim that this is their goal is unsettling. This view is perhaps most succinctly stated by the startup Mechanize, which aims to “fully automate the economy”. As a participant in the economy, I find this goal deeply objectionable. I think that trying to fully replace all human labor is not a morally acceptable goal to have. It’s frankly horrifying. To be clear, “fully automate the economy” means “destroy society as we know it”. Unfortunately, similar goals are articulated by quite a few inside and outside Silicon Valley. I think the callousness of such objectives should be called out whenever they are encountered.

The great irony of this is that those who claim to want to automate all human labor are typically the kind of competitive, smart, high-agency persons who absolutely need to have a purpose and something to build. They would hate to be redundant. Yet, here they fly, like so many moths to a flame.

Yes, I have also said in the past that I wanted to develop AI systems that can do anything a human can do. But that was in the past. Back when AI did not work well, and few seemed to have a vision for going beyond specific, well-defined applications.

I guess you can see now why I am having a crisis of faith. I used to believe that deeper understanding of the world and greater technological capability was, in a sense, our purpose. But our greater technological capability might lead us to a world where we can no longer make a difference, and there is little point in us understanding more. Perhaps we get abundance, but at the price of redundance. Again: nobody actually wants this. And I, a mere AI researcher, am no longer sure about what to do. The thought of simply quitting the field has occurred to me, but I’m not sure that would make the world better, or even exorcise my personal demons. Besides, AI research is the one thing I know how to do well. And I need that paycheck.

So I’m left with the Cartesian task of trying to reconstruct my belief system, starting from what I really value.

Well, I believe in humans. I believe in individual human brilliance and creativity. But I also believe in society. I believe that we matter for each other, and that we must matter for each other. The world I want is one where we can all fulfil our individual potential, but where we need and admire each other.

First of all, throughout all of history we humans have adapted to and mastered the technologies we invent. Some may say we have merged with our technologies. We are so dependent on our technologies that the whole idea of us humans ourselves having general intelligence only really makes sense in the context of all the tech that we have embedded ourselves in. We feel so intelligent because we have built a world that empowers us so much.

So far, so good. But is not this new technology we call artificial intelligence fundamentally different from writing, steam engines, electricity, and telephones? It is. But these technologies were all very different from each other. Yet, we adapted to each of them. We learned how to build with them. Completely new domains of human excellence appeared that we couldn’t even imagine before new technology had made them possible. Because cars were invented, you could become great at designing, repairing, selling, racing, or writing about them. In 2025, you can also shine by streaming your Minecraft creations on TikTok. Who knows what new things you can be good at in 2045?

Then, we can note that there is a lot of scientific and technological progress that we actually want. I want us to be able to travel to the stars, cure all diseases, and make better video games. There are also so many things we don't know yet - every academic field has more questions than answers. The world is full of wonder. So let’s discover and invent all the things they have in Star Trek, minus the murderous aliens. But let's make sure we do the inventing and discovering, with the help of the machines. (It is also worth noting that Star Trek is so good because it always centers human agency and cooperation, which the showrunners accomplish partly by limiting what the computers can do.)

You may have noted that I am placing myself in a long tradition of people worrying about how new technologies will dehumanize us and ruin society someway. Your eyes may have been rolling over several times as you read this text. Basically every technology of note - from writing to telephones to video recorders - have come accompanied by commenters warning about how they will somehow end the world as we know it. Why would thinking machines be any different? I can only say that I hope that you are right and I am wrong. I really do. I want nothing more than returning to the optimism for the future I used to have.

When I'm in particular need of optimism, I look at my young children. They will come of age in a world different from mine. To them, whatever machine capabilities they grew up with will seem perfectly natural, just as desktop computing and early computer networks seemed to me. I hope they will have opportunities to contribute to society by doing something they are really good at, better than the machines. If not, they will barely have known any other world. Above all, I hope they will be happy.

Redefining my religion, summer 2026

I wrote the text above just over a year ago. Clearly, AI capabilities have not stopped advancing since. Quite the opposite. So how can I be less worried today?

Well, first I started going to a therapist. We were not a match, so I moved on and found me a psychiatrist, who I got on much better with. He prescribed me an SSRI, which has helped me get the peace of mind to focus my thinking. One thing we puzzled over together was why these worries gripped me now, given that the thoughts themselves are by no means new to me. I’ve worked on AI all my adult life, read boatloads of science fiction, and have a degree in philosophy. I remember thinking about the singularity and intelligence explosion concepts as an undergraduate in my early 20s, and how I already back then thought that the assumptions made in those arguments were so simplistic as to make them essentially meaningless. For most of my adult life, I’ve been an even-keeled person with a mostly sunny disposition. So why the anxieties now?

The triggering event in my case was probably an episode in 2024, where I was provisionally diagnosed with a very serious disease. The diagnosis turned out to be wrong, but for almost two months I lived with a fear of death that blanketed all my waking moments. When I laughed, it felt hollow. Moments of happiness were drained of meaning because of the ubiquitous thought that life could soon be over and all this be gone. I very distinctly remember standing outside the Mount Sinai hospital in Manhattan after learning that I did not have the disease they first thought I had, and that I was probably going to live a long and healthy life after all. I had no idea what to do with all my emotions. My psychiatrist suggested that this monthslong episode had awakened the fears I used to have as a child, and the worrying predisposition I had back then. Now that I was no longer acutely fearing death from disease, my fears needed something new to latch on to, and it was right there in front of me: artificial intelligence. The happy and stable person I had been all my adult life had turned into a worryer for mostly incidental reasons.

My psychiatrist also suggested, half jokingly, that it was useful for me to get my existential crisis in early, so I would be ready to help when others get to this stage. Not a bad idea, I thought. That’s why I’m writing this.

Let us talk about perspective. I have found that putting my worries in perspective is great for my sanity. Most people I know do not worry about AI rendering us intellectually irrelevant, although some do. Some do not really worry very much, others worry, but about other things. A close friend of mine is consumed by worry about climate change. Many others worry about the rise of fascism and populism, and the risk that the US and perhaps other countries turn into full-fledged autocracies. Some worry about war, nuclear and conventional.

I don’t mean to belittle these concerns. Climate change is real and is going to cost the whole world dearly. The political situation is not great in much of the world, and it’s not trending the right way. As a kid, I was in constant fear of nuclear war, and those nukes are still there. There are multiple wars around the world, including a major land war in Europe. Still, none of these very real issues grip me emotionally in the way intellectual obsolescence because of AI does. The latter is the only one that can keep me from sleeping.

Reading this, you might think that I am being ridiculous. How can I worry about AI making us intellectually redundant, but not worry as much about fascism, war, or global warming? You may also worry about something completely different that I haven’t even considered worrying about, such as the dominance of some particular ethnic or religious group, or vaccine conspiracies, or aliens. Or for that matter, you may worry that a superintelligent AI will break free and kill us all, which I also don't really worry about. The point here is perspective. Why do I worry about this particular thing, and not all the other things that other people worry about? There are so many things I could potentially worry about, and worrying all the time is clearly no way to live. That I‘m stuck on this particular issue and not something else is probably because I work in AI and drink from the firehose of AI discourse all day. I know, it’s not healthy, which is why I’ve made efforts to diversify my information diet.

I've found it particularly useful to read plenty of history, in particular on the various upheavals that have followed intellectual and technological change. Like with all of history, it is not always uplifting reading. But it does drive home the points that society changes at human pace even when technology changes much more rapidly (as also argued in e.g. Narayanan and Kapoor's excellent "AI as Normal Technology"), and that humans always keep coming up with new things to do. Also, for what it's worth, these changes tend to result in interesting art.

Why didn't I just quit AI research? My predicament would seem like that of a vegan butcher, or a monk who makes money on Onlyfans. But me quitting and becoming an Uber driver would not make the world better. The pace of AI progress would clearly not slow down noticeably. And I assure you, I still love AI research, even if I sometimes hate what AI does to the world. I'm not even very good at anything else. AI is what I do. And I think that I can do more good by trying to steer my field in a good direction than if I became an Uber driver. So I’d rather think of myself as akin to a hypochondriac doctor, or a pilot with a fear of heights.

What about my religion? Did I find my way back to it? Yes, but after a complete update to version 2.0. I still believe that the progress of science and technology is deeply meaningful. But it has become clear to me that this must be research done and technology invented by humans, for humans, and understood by humans. If humans are not in the loop and at the steering wheel, it is not progress. Of course we should use technology to help us, as we have always done, but it must be we who use the technology rather than the other way around. It does not matter how amazing progress we make if we are not the ones who make it.

To some of you readers, the paragraph above might seem trivial, almost like a tautology. Did it really take me 47 years to understand something so obvious as that science and technology must be for humans, by humans? Some kind of researcher I am. To other readers, I am just plain wrong. For example, at NeurIPS 2025, I voiced some critical concerns at a panel in AI for science. I wrote up these comments as a post called "please, don't automate science" which briefly made me Twitter famous in the wrong way. Basically, what I said was that cutting humans out of science would be terrible for humanity, no matter how much research could be sped up. Several hundred people (well, several hundred accounts) accused me of being wrong, stupid, and/or evil. Many thought that it was unavoidable that humans would be automated out of science. Some even thought it was desirable! To my critics, faster technological progress would be worth giving up all agency and even understanding. To them, I am anti-progress. To me, they are technologically defeatist or even anti-human.

I am in some sense a man of action, although action in my case often consists in typing words into a document. My parents, who are artists (I said this earlier, but the text is long and it might have fallen out of your context window by now) told me that the way they handle their fears is by making art out of it. So the best thing I can do about what scares me is the thing I know how to do: AI research, starting with the theoretical framework.

To that end, I've developed my thinking around what I call complementary intelligence. This is a continuation of a perspective I've developed over a while and written about in various places, including my Artificial General Intelligence book. The basic tenet is that there is not a single thing called intelligence; there is a highly complex set of behaviors and capabilities that have evolved to fit the particular ecological niche of each species. When it comes to humans, we have built our own niche; most of the things most of us interact with every day are designed by us, for us, to amplify our agency. That's why we think we are so intelligent.

From this perspective, the history of AI is a history of attempts to mimic the specific combination of behaviors and capabilities that humans have; most of them successful in some way, but all of them quite different to humans. The onslaught of LLMs becomes a push in a particular capability direction. Understanding that direction becomes crucial to figuring out which types of human intellectual patterns and capabilities will become more important in the future. Where the new domains of human excellence will appear. But this understanding can also help us develop different types of AI that are more complementary to what humans can do and like to do. Seeing intelligence as a scalar, where machines can overtake humans, is a recipe for paralysis; dissolving this faulty notion gives us the freedom to act. More of this argument in the article I linked above; what's important here is that there are things that can be done. Indeed, there's a lot to do. Such as building mechanisms for meaningfully incorporating humans in creative search processes, and open-ended learning and discovery processes that are not based on imitating what humans do.

But not everything has a technical fix. Norms, structures, and laws are probably more important. I still think that trying to automate humans out of the processes that give them, and our civilization, meaning is immoral. We need to build counter-narratives, and be vocal that "fully automating the economy" is not an acceptable goal to work towards. We also need to push hard to counter the centralization of power that so easily comes with lavishly funded tech companies trying to achieve monopolies on some layers of the AI stack. Our best bet for a future where we all matter is one with a myriad different forms of intelligence, open and accessible for all to use as tools for our natural intelligences. Let's get to work.

What if

What if the superpower of the future is self-discipline. The discipline to write enough code yourself to understand coding well enough to effectively outsource coding to agents. The discipline to learn math by proving theorems yourself. The discipline to write your own texts, so you learn to think. The discipline to pay attention. The discipline to sit with emotionally uncomfortable situations until you understand those emotions, yours and the other's. The discipline to read old long books. The discipline to repeatedly bang your head against the wall. The discipline to be bored. So that you can think beyond the obvious. Beyond the next token.

What if the way to become an AI super-user is to learn to resist AI almost all the time. 

What if the way to win at Chess is not to move the pieces as fast as you can. What if the way to find true love is not to enter a new relationship every week. What if Pythia only allowed you to ask one question.

What if the schools of the future require you to sit straight and write in your book with a graphite pencil, in total silence, for hours at a time. What if the mark of a successful person in that future is an unhurried demeanor and a steady gaze, lending you their full attention for as long as you deserve it. What if the capital of the future is not San Francisco.

Congratulations, you have reached the end of this post. Feel free to keep scrolling.

Complementary Intelligence

 In the following, I will try sketch a way of thinking about human intelligence and human nature through emphasizing its difference from the various methods and systems we call artificial intelligence. This is rooted in my strong belief that talking about a single-dimensional “intelligence” that one can have more or less of, and the obvious extension to asking whether humans or machines are “more intelligent”, is actively harmful for understanding both human and machine intelligence. What matters is the qualitative difference. Between humans and machines, but also between different approaches to AI. We can even think of the different approaches in a geometric framework, with the implication that any type of intelligence must have a direction as well as a magnitude. This perspective, which I call Complementary Intelligence, also suggests a positive research program, that seeks to find the types of intelligence that we do not currently have but that could be interesting to humans, rather than simply imitating human intelligence.

Ok, this was a lot. Let’s rewind the tape, and start by looking at the history of artificial intelligence and the ways we think about it.

You can understand the human mind by looking at the history of our failures at modeling it. For a very long time, we have tried to make machines in our image. After we invented the digital computer, this development sped up and we called it Artificial Intelligence. During the last 70 years or so, the AI research community invented a number of clever ways to make computers do things that so far only humans could. Usually, we set ourselves some problem – play Chess, prove mathematical theorems, translate from Russian to English, or something like that - that humans were good at. Then, we came up with some way of making a computer perform the task. Success!

But when we look closer, we find that the way the computer does the task is typically quite different from how humans do it. The computer may be much better than humans in some ways, and much worse in other ways, and in general just different. So we conclude that we didn’t really achieve “real” artificial intelligence after all. Maybe we were trying to solve the wrong task? So we find another task to solve, and another way of making the computer solve it, and try again. As a result, the study of artificial intelligence has contributed a wide range of technologies, many of which are crucial to our technological civilization. In our urge to talk about AI as a single thing, it is often underappreciated how many these technologies are, and also how different they are. Path finding, object-oriented programming, and optical character recognition are quite different things, but they are all outcomes of AI research. They are also in use in myriads of places and the world would grind to a halt if they disappeared.

Another result of this process is that we know more about what we are not. Every time we realize that the successful solution we have built is fundamentally different than us, we learn something about ourselves. We learn that we are not like that technology, or not only like that technology. However good that machine is at identifying traffic signs or playing Pac-Man, it does so in a fundamentally different way than we do it.

In a sense, this is just a continuation of our long history of using the defining technology of whatever age we are in as a metaphor or lens for understanding ourselves. Descartes, living in an age recently transformed by the mechanical measurement of time, thought of animals and humans as being like clockworks. Freud thought of our drives as producing something like pneumatic pressure requiring outlets, much like the steam engines that pulled trains and powered factories. The telephone switchboard was a popular metaphor in early 20th century neuroscience. But of course, if you try to actually build a mind that functions like a clockwork, a steam engine, or a telephone switchboard, you rapidly realize that there’s a lot missing. And from the incompleteness of the metaphor, you conclude that we are much more, and quite different.

Let us therefore try to sketch a history of AI focusing on not only its successes, but its complement: what we have learned about what we are not. This will by necessity be a very potted history.

The earliest successes of what is now known as AI were based on planning. This includes early Chess and Checkers players, and automated theorem provers such as Logic Theorist. The basic idea is to start at some state (such as a board position in Chess, or an axiom from which you want to derive a theorem) and consider the various possible actions available from there (moves in Chess, transformations in theorem proving). As considering all possible consequences of all possible actions recursively becomes computationally intractable for all but trivial problems, much of the art of planning is in the heuristics for which actions to consider at each point. Already in the 1950s we had theorem proving systems that could rediscover some previously discovered theorems much faster than humans, and in the next decades we saw major successes. In 1996 the Robbins conjecture, a long-open problem, was proven by a search-based theorem prover. Similarly, planning approaches led to superhuman play in classic board games such as Checkers and Chess.

In particular for board games, there has been quite a bit of work comparing humans to planning algorithms for game play. It turns out we are not alike. The algorithms explore many, many more potential moves and board states. Humans tend to explore just a few move sequences, but be much better at evaluating the positions.

In the 1970s and 1980s, expert systems were one of the main foci of AI research. The idea was to encode the knowledge of human experts in a form amenable to logical reasoning, and then let the computer do the reasoning rather than the human expert. Alas, it was not so easy.

Extracting the requisite knowledge from human experts turned out to be an enormous time sink. Humans, experts or not, seemed to have a hard time expressing what they knew. In particular, they found it very hard to express procedural knowledge (how to do things) in a way that could be formalized as rules. The finished systems often turned out to be brittle and inflexible, needing humans to look through decisions, which obviously severely limits the usefulness of the system. An often-used example is MYCIN, which was developed at Stanford in the early 70s to diagnose blood diseases. It took years to encode the 500 or so rules that the system used, and despite good performance in trials, MYCIN was never used in clinical practice.

The most reasonable explanation for the limited success of expert systems is that humans do not store their knowledge as a set of logical statements and rules. This might seem like a pretty obvious thing. Did anyone ever think that our brains operated this way? Surprisingly, yes. A long history of thought–ever since Aristotle–has postulated logics not just as a normative ideal about how we should think, but as a theory of how we actually think. More explicitly, the computer metaphor of the mind that become popular with the rise of cognitive psychology and the advent of cognitive science explicitly compares the functioning of the human mind to a standard computer, von Neumann architecture and all. The best interpretation of the relative failure of classic expert systems is that the computer metaphor cannot literally be true, at least at the level of how we encode knowledge.

Neural networks in one form or another underlie almost all modern AI. That much is generally known. Less often mentioned is that the earliest computer models of neural networks were proposed back in the 1940s, and the backpropagation algorithm that is the direct predecessor of the optimizers used in modern deep learning was invented in the 1970s. While there have been numerous minor and medium-size inventions in neural networks since, the remarkable success of the neural network approach is to a large extent due to us having more data and more compute so we can train larger networks. 

Much has been said about how neural networks mimic some features of human learning, such as learning hierarchies of representations. Less is said about how profoundly different they are to human brains. To begin with, there is no evidence of anything like backpropagation going on in the brain. This is reflected in how differently neural networks learn. In most settings, a neural network must see many more training examples than a human to learn the same concept. And when a concept is learned, it seems to be brittle. For any given model, it seems to be possible to find “attacks”, where changing a few tiny elements of the image completely throws the neural network, making it classify a panda as a gibbon or an abstract pattern of yellow and black as a school bus. Modern foundation models keep being susceptible to jailbreaks and prompt injection attacks. For all their proficiency at recognizing patterns, neural networks clearly do this in a different way than we do.

Similar things can be said about reinforcement learning. Seemingly miraculously, we can train neural networks to play games or control robots based only on feedback on their behavior. It’s astonishing that this works at all; it’s essentially trial-and-error on a massive scale. But why is such massive scale necessary? DeepMind’s classic experiments on learning to play Atari games with deep reinforcement learning saw each game being played for an equivalent of 38 days of game time. In contrast, a human can usually learn to play such games in less than an hour, sometimes in mere minutes. Of course, humans do this partly based on their familiarity with other games, as well as a lifetime of learning other visuomotor skills, from hopscotch to chopping onions. Artificial reinforcement learning systems are not good at this. Typically, they struggle to generalize beyond the narrow setting they have been trained on. Those networks that spent 38 simulated days to learn a simple Atari game? If you make a tiny change to the game, such as remapping the colors, or changing a few pixels here and there, they become utterly helpless.

There are other paradigms within AI that contextualize our own intelligence in other ways. Such as evolutionary computation. By (often crudely) mimicking Darwinian evolution, we can solve a large variety of problems. Evolution can come up with new designs for antennas, surprising but lucrative trading portfolios, useful software, and many other things. Evolution can also be used for supervised learning and reinforcement learning, often with more or less equally good results as the more commonly used gradient descent methods. But isn’t this weird? How can we get such good results through a completely different type of algorithm? Clearly, the currently dominant paradigm of AI is not the only way of solving the various problems we use AI to solve. 

The viability of evolutionary computation also reminds us about the perhaps greatest product of natural evolution: us. We are evolved beings, with an evolved culture. The main reason that we perceive ourselves as being generally intelligent is that we have built a world tailored to our shared cognitive capabilities. These capabilities have evolved over hundreds of millions of years. When we come to the world and start thinking, we are not blank slates; we build on an intricate neurophysiology and a vast repertoire of skills, instincts, and perspectives, some of which might at some point have helped our ancestors pick non-poisonous fruit, outwit crocodiles, or predict when the rain would come. In contrast, a machine learning model is quite literally a blank slate before training starts. Or rather, a blank matrix. Unlike all AI in existence, we are “trained” in a multi-timescale distributed process, encompassing our whole phylogenetic lineage as well as our whole culture.

Which brings us to present day. We now have large language models, and they are like the mind of god. At least according to breathless hypesters and accelerationists. More sober commentators still recognize that they are some of the most impressive technology we have ever seen, and they may well turn out to be almost uniquely consequential. We have all been humbled by LLMs doing something we didn’t think they could. Some of us multiple times. What are their shortcomings?

To begin with, they are good at tasks largely in proportion to how easily these tasks can be represented as strings. If the input is text and the output is text, chances are the LLM can solve the task very well. Modern multimodal models are also now very good at generating and classifying images, which are internally represented as strings of tokens. But spatial reasoning and interaction is another matter. Currently, huge resources are spent on trying to make these models confidently interact with graphical user interfaces. Granted, they are getting better at it. But they are still atrociously bad at, for example, playing video games. (Unless the game is very well known and you build an elaborate harness for it.)

It is likely that multimodal models will soon get much better at spatial interaction, at least for tasks that are economically relevant. The bigger issue is the lack of memory and continual learning. The current state of LLM memory is like the protagonist of the movie Memento, an amnesic man who can’t form new memories, and therefore has to write little notes to himself (or tattoo notes on his body) to remind himself who he is and what he is doing. This is because an LLM does not modify its parameters as you interact with it. All the little numbers that define it remain frozen in time. Instead, it keeps a short-term memory of its interaction in its context, but the length of this context is necessarily limited. To achieve something akin to long term memory, the harness around the LLM will at intervals summarize its context as a text file and store it away in a kind of database, which it can then access in the future. Rather like writing little notes to itself. This is likely to be a fundamental limitation of LLMs, not in the sense that it cannot be overcome, but in the sense that the solution will look quite different to an LLM as we know them today.

There are other ways in which LLMs differ from us which have not yet understood fully, because the technology is so new. For example, just like other forms machine learning, LLMs appear to have a strong bias towards problems that are in its training set. One way this manifests is a curious lack of novel insights stemming from LLMs recombining existing knowledge. Very much, if not most, of human creativity comes from recombining existing knowledge. Now, LLMs have a broader range of “expertise” than any human ever. Which actual human would simultaneously have detailed knowledge of peat bogs, pupillometry, polymerization, Pasadena, and Paul Krugman? In fact, frontier LLMs have had this staggering breadth of knowledge for at least 3 years, since GPT-4. A human with such range would surely make a stream of unexpected connections. Yet, few if any truly novel insights are directly attributable to LLMs. Why? We don’t know. We also don’t know whether this is a fundamental limitation of this approach to AI.

So far, we have only talked about intelligence in a relatively abstract information-processing sense. But we are not just brains, we are whole bodies. As you may have noticed, the way you think is strongly affected by whether you are hungry, horny, angry, or something else. And much of your thinking involves your body in some way, whether it is walking, tying your shoelaces, or typing on a keyboard. Some argue that all of your thinking is rooted in your body. Opinions diverge within cognitive science as to how important the body is to thinking. But what is plain to see is that physical robots are far, far behind non-embodied AI. Robots struggle to do things that are trivial for us, such as opening door handles. This is not a new issue: it has been the case for the whole history of AI, and much commented on.

Replaying the history of AI this way, we can sketch a different understanding of human intelligence and human nature than what we would get from using the AI we built as a metaphor for ourselves. More precisely, we can paint a picture of intelligence that emphasizes the parts which our AI systems are not good at, or which they do in a very different way to us. We can emphasize the complementary part.

Let us consider the difference. If we did use AI as a metaphor for our own intelligence, or as a lens for understanding it, similar to how previous generations used clockworks or steam power, we would arrive at what could be described as a rather classicist picture. Human intelligence operates by considering a large range of alternatives, tries to solve specific tasks that have well-defined rewards and can be clearly separated from other tasks, learns each skill on its own, starts from a blank slate when learning, and sees task descriptions and world descriptions largely as text. This picture has echoes not only of philosophies of past centuries, but also of modern management thinking and the kind of postmodern thinking which sees everything as a “text”.

The complementary intelligence view is instead that we are creatures that are deeply rooted in our history, both our evolutionary history as a species, our cultural history, and our personal history. Context is what we excel at. Most of what we do cannot easily be stated as separate tasks with well-defined rewards. We learn and reason slowly in terms of clock time, but effectively in terms of number of examples we need to see. We almost never operate according to logical rules, though we may tack them on as justification for what we did. Text is just one of our modalities, and somewhat “tacked on” compared to for example sight, smell, or proprioception. The body plays an important role in our thinking, and fine manipulation is another thing we excel at.

Neither human nor artificial intelligence is “general” in anything but a trivial sense, and could never be. The reason we believe we have general intelligence is that we live in a world we have constructed over the course of our civilization to fit our capabilities perfectly. Our societies, technologies, and built environments are scaffolding and support systems for our very particular type of intelligence. This makes us feel very smart and powerful. But thinking that the particular capabilities that the world we constructed test and amplify is all there is to intelligence is a very parochial view.

Looking at human intelligence this way gives us perspective on the rapidly advancing capabilities of AI. It is often asked when AI will overtake human intelligence. But this assumes that intelligence is a single-dimensional quantity. The various types of machine intelligence we have created can instead be seen as vectors pointing in different directions. Classic symbolic planning is one vector, LLMs are another, and fuzzy logic also another. Human intelligence is yet another direction. Moving further along in one direction (increasing the magnitude of the vector) may have limited bearing when projected on other intelligence vectors.

So, to directly answer the question about when artificial intelligence will surpass human intelligence: it did so long time ago, many times, and it never will. Various technologies that we refer to as artificial intelligence have surpassed humans at calculating, planning, solving logical puzzles, factual recall, and many other things. Yet, it is extremely unlikely that any technology would have exactly the same intelligence vector as human intelligence. Because these machines are not humans, their intelligence will always point in different directions.

Interestingly, the history of AI can be seen as a sequence of attempts at approximating the human intelligence vector. The moving goalpost phenomenon then becomes a game of finding a particular point on this vector, only describing it in one or a few dimensions, and trying to invent something that reaches this point. We then reach that point, only to discover that we did so by following a completely different vector than then we tried to imitate. So we find a new point, and repeat the procedure.

We can use complementary intelligence as a term to describe this view, but we can also see it as a positive research program. Complementary intelligence as a direction is, basically, to lean into the difference. We should not try to eliminate it, instead we should support it. Recognize the strengths of the human intelligence vector and build AI systems that amplify it.

At the same time, we should try to move away from trying to approximate the human intelligence vector. For example, plenty of AI researchers around the world are currently working on how to augment LLMs with continual learning, because they realize that’s not something that will be found along the intelligence vector of current LLMs. I think we should take our efforts elsewhere. Simply imitating human capabilities is perhaps the least interesting way of building artificial intelligence. It’s so unimaginative. And it leaves so many potential capabilities on the table. You could even argue that fully imitating human intelligence is immoral. We don’t actually want artificial intelligence that has all the capabilities we have, and is better at all of them. Because we want to matter. And we don’t want to be replaced.

Instead we should seek to amplify machines’ capabilities at tasks that we humans are not particularly good at, or do not want to do. In the best case, tasks that are not done at all, because we can’t or won’t do them, even though we might want to. We want AI that lets us focus on the things that we want to be good at, and give us abilities we didn’t have before.

To take some very quotidian examples: high-frequency trading is an example of complementary intelligence, because we can’t trade that fast. It is literally physically impossible for humans. AI deployed inside a video game, to generate levels, control non-player characters, or something like that, is another example of complementary intelligence, because you could not have humans implementing every NPC, and they would probably be very bored if they were asked to follow the rules that NPCs follow. AI methods for helping us make sense of modalities we are not attuned to, from WiFi reflection to gravitation fields are also good examples, as they expand our perceptual space. Then there is of course the boring but immensely impactful technologies that make the modern world possible and does not replace any cognitive work people actually want to do, such as databases and web search.

But beyond this, there is a virtual infinity of new types of intelligence we could develop, and new tasks we could discover, and new solutions we could invent. Taking the metaphor of intelligence vectors seriously, we could envision a hypersphere of capabilities on which every possible intelligence vector, at its maximum magnitude, could only reach a particular point. By definition, we have only explored an infinitesimal part of the interior volume of this hypersphere. There is so much more to do.

Most of these types of intelligence would likely be uninteresting and indeed incomprehensible to us humans and our society. But there is likely to be a practically infinite number of directions we could appreciate and build exciting new capabilities around if we first invented them. I don’t know what these capabilities would be, because they have not been invented. But I think the semi-automated open-ended search for new types of intelligence and associated tasks that would likely be of interest to humans to be the most exciting direction for AI I can imagine. This will definitely require new thinking about open-ended search, discovery of new ways of measuring search space, and clever measures of what humans find interesting.

As you can tell, there’s a lot to be worked out here. I’m thinking I should write my next book about Complementary Intelligence, so I get a chance to work some of it out. What do you think, should I?

Computers and me

I read things, write things, and talk to people. The proportions vary, but that’s essentially what I do. Or rather: those are my observable activities. I also think. The thinking often happens when I read, write, or talk, but also when I walk, drive, or take the subway or elevator. And when I shower! That exclamation mark! I should shower more.

Once upon a time I also programmed. I even considered that my craft, on par with writing. The last time I wrote non-trivial code was 2015. Long before that, I’d stopped keeping up with modern toolchains and software development practices, or even languages. That’s actually partly why I stopped programming: nobody uses Java for AI research or SVN for version control, and I think Python is unacceptably sloppy and git is incomprehensible.

Of course, the main reason I don’t program anymore is that I’m busy reading, writing, and talking. And thinking. I enjoy those things more. It’s not that I didn’t like programming: I enjoyed it a lot. And I was quite good at it. But there are lots of enjoyable activities you don’t easily find time for when you have two jobs and two kids. Even activities you’re good at.

Before I programmed, I took things apart. First, my toys. My room was full of useless thingamagogs that had once been part of fully functioning toys. I was no good at putting them back together again, or I didn’t have the patience. Or the interest. At some point, I graduated to computers, and built various PCs from parts I bought cheaply from flea markets or badgered my mom’s friends to give me. I destroyed a lot of those parts in the process. It was a lot of fun.

The PC-XT clone I bought with the proceeds from my first summer job (as a gardener) when I was 13 had a Turbo Pascal IDE on its 20 Mb hard drive. I decided to learn to program so I could make games. I copied and pasted things and tried to figure out what worked through trial and error. I learned a thing or two. Later on, I also spent a lot of time composing music on a 486 I built myself, and learned the basics of website building on the same machine. I never even fastened the hard drive to the chassis, and the computer had blinking lights and some kind of glitch so that you might get an electric shock from touching it.

These days, I don’t want to see the insides of my computers. I use Macs, and I want them pristine. No stickers, clean desktop, and no unnecessary applications. As few customizations as possible. It’s like I’m not even interested in computers anymore.

In sum, I’m a bad computer user. I do not let my computers fulfill their potential. Basically, I use the computer for reading and writing. Anything I actually use my computer for could be done on a 20 year old machine. If it could connect to the internet, I could do what I do on a 40 year old computer.

Yet, I keep buying new computers. I happily hand over my employers’ money to Apple in exchange for swanky new gear with waaaay more power than I need. And I don’t feel bad about it. I tell myself that I need an M5 Max with max memory so I can run local LLMs, and that is in fact a minor hobby of mine, but not really important to my actual work. Most of the time I use my computer for reading and writing emails, or reading papers or web pages, or having Zoom calls. My jacked monster of a swole M5 processor must be really bored.

I think I like computers mostly for aesthetic reasons. I’m like a rich old man who buys a Ferrari only to drive it around town and never exceed the speed limit. I just want to hear the menacing growl of the V8 and admire those aerodynamic lines. Except I’m not rich, and not that old, so I buy computers instead.

I’ve been thinking about this recently because computers are finally learning to use computers. Fat harnesses around frontier models help them navigate various applications, and this means you can increasingly just ask your computer to do things for you. Language models can also write code really well now, so you can (sometimes) conjure functioning new software just by calling it by its true name. Thus, it’s all the rage to make your AI agents do things for you. Writing code, reading reports, answering emails, other computer things.

Some seem to want to automate all of their digital life. Some seem to think it’s a good idea to install OpenClaw and give it root access to their computer and logins to all their computers. These kinds of people remind me of myself when I was 16, deeply into building weird things that rarely worked just for the sake of it, customizing every piece of software and interface because it’s cool, and caring not for safety nor security. I try to keep in mind that I was also once like that, because that allows me to understand these people. They just love technology in the way I once did.

Anyway. I am allegedly an “AI researcher”, a type of “computer scientist", and this comes, I think, with the obligation to at least occasionally act like one. So I try to use all these frontier models like I was Buffalo Bill. Often, it involves looking hard for some need I barely have that might be satisfied by a language model. This task is getting harder and harder. For what do I actually need? What should I use these things for?

A friend of mine suggested I vibe-code some unique software just for me. What kind of software, I asked. He said he had made some software for himself that keeps track of his exercise routine just the way he wants it. But I don’t want that! The point of going to the gym is to not have to care about such things, and instead put on the headphones and zone out while incinerating calories and letting the mind wander. Also, I don’t want to have to take care of maintaining a piece of software, even if it’s just for myself. Unnecessary stress. In fact, I want less software, not more. There’s far too much software in the world already. For any given need, there’s probably an app for that already, but I don’t want to have to look for it and I don’t want to install even more apps. That my local lunch restaurant has its own app and pesters me to install it is proof that too much software is being written.

What else could I have the models do for me? Write for me? But the whole point of my writing is that it’s mine. It’s not so much that it would be immoral to put my own name to something an LLM wrote (it certainly would), but that it doesn’t even make sense. It just wouldn’t be my writing. Please don’t tell me I need to explain this to you.

Could I have the models think for me? But I thought we already established that I am in this job because I like thinking. If you want to avoid thinking, you should not become an academic. Imagine going to a restaurant, ordering food, and then paying extra for the waiter to eat the food as well. That’s right, eating the food yourself is kind of the point. (My original metaphor was more striking, but this is a family-friendly blog.)

The more I think about it, the more the advent of AI agents has made me realize that I’m not much of a computer user. I don’t care for the vast majority of things you could make a computer do, and I don’t want to bother with new software if I can avoid it. Please don’t bother me with your buzzwordladen productivity catalyst. Give me a text editor and shut up already.

Maybe I would rather not even use computers. The computers can use themselves now, so maybe we can get on with our lives? Computers aren’t real anyway. So let me valet my shiny laptop. What I really need is some good books, some good friends, and a typewriter. And some good wine. So I can read, write, talk. And think.