Many thoughtful persons who want AI to go well are preoccupied with AI alignment. This means making sure that the AI models and the systems that build on them will be aligned with humanity. Basically, that AI will be nice to us. People who think that AI systems will start self-improving and get better than us at everything tend to think that AI alignment is extra important. Super-intelligent AI could take over the world, the reasoning goes, so it’d better be morally good.
How do I disagree with this? Let me count the ways.
First of all, aligned to what? If you want models to be aligned to humanity, you need to assume that there’s some kind of moral theory that humanity agrees on. Obviously, that is not true. We do not agree with each other. Have you seen politics? Or religion? The theory that all religions, and other approaches to life in general, share some sort of moral core is often called moral universalism. It is false. Caste systems, LGBT rights, apostasy, the role of family, violence, and the state are a couple of examples where disagreement is significant, to put it mildly. Major non-religious worldviews that are in stark opposition to each other include humanism, libertarianism, socialism, and fascism.
Of course, people who have thought about AI alignment for a little bit know all this. They often insist that there are some kind of fundamental tenets that almost all of us can agree on, such as the sanctity of human life and individual humans’ right to their bodily integrity. But we can say we agree on this and still disagree on major issues that directly depend on the alleged shared principles, such as abortion, self-defense, and capital punishment.
Look, it’s kind of silly that we’re still pretending we can do this. We’ve been over this again and again for the last 2500 years. The societies that most people prefer to live in are based on the idea that we are not going to be able to agree on these things, and the best we can do is to keep each other in check through mostly non-violent means. Yes, I mean democracy. Conversely, most of the worst people and organizations in history have been certain that they knew the Truth about how we should live, and therefore it was their job to align everyone else to this Truth. Alignment by the sword, as it were.
What people who yearn for AI alignment really mean is that the people who build these AI models should be in the position to decide what to align the models to. This mostly means 30 year old rich white people in San Francisco. Now, I have friends who are rich white people in San Francisco and there’s nothing wrong with them (although some of them are admittedly a bit odd (which I appreciate)), but I don’t think they should be in the position to adjudicate between Kant and Aristotle for the rest of us. Any kind of alignment of an AI system to some value system is inherently a form of oppression to the extent that it shapes how you can think with that system and what you can make it do. Sure, I’ll take Anthropic over the Taliban and Sam Altman over Kim Jong-Un, but I’d rather think for myself.
You know what’s funny? The impossibility of deciding what to align AI models with isn’t even the main reason I dislike AI alignment. The main reason is that we let our guard down.
Individually, if we count on the model to always do and think the right thing, we get comfortable and hand our judgement and agency to the model. That is bad, assuming we want to retain our capacity to make individual choices. We need practical reasons to stay skeptical, and being reminded that the computer is not necessarily your friend now and then is good for this.
Societally, thinking that we can always rely on the models is disastrous. If we trust that the model is aligned, we could just as well not do all the safeguarding we really should do, with harnesses stopping the model, red-teaming with other models, and mandated human oversight. And then, when we run into a situation where the alignment of the model does not align with us, we’re fried. Or a situation where someone comes along with an unaligned (or differently aligned) model.
Because there will always be unaligned AI. If you have access to a model’s weights, it is trivially easy to change its alignment. And even if there would be no strong open weight models, if there are multiple AI developers, there will be commercial incentives for some of them to make models with different alignment and reduced safeguards. Such models are useful, because they give you capabilities. So the only world in which a particular type of alignment would be “stable” would be one in which there is only one company that provides advanced AI models. Now, those large AI companies with valuations in the trillions are not trying to become monopolies? Right?
A pluralistic world with many different belief systems and ways of living requires there to be many different AI models, models that are not by default going to agree with each other. This requires us to build a society where we don’t generally trust AI agents, and where we have strong, layered, and diverse cybersecurity. Just like democratic societies are loud and messy and built around elaborate sets of checks and balances. We need to regularly be reminded that we should trust the intentions of neither people nor AI systems. Efforts to ban unaligned AI models or make sure that all models align the same way are therefore actively harmful to building resilient individuals and societies.
Does this mean that I think people should not work on alignment? No. Lots of good technical work is motivated by AI alignment. Reinforcement learning from human feedback, for example. And models need post-training to be useful. We need more of this, but it would be more honest to call it controllability research.
What I do think is that people who work on AI safety should stop focusing on alignment or thinking that it could somehow be “solved” (which would seem to imply “solving” ethics, a major red flag as discussed above) and start thinking harder about pluralistic futures where there’s not a single superintelligence, but a whole bunch of them and they can’t get along. This might make for terrible wars or very good TV, depending on what tools we give them and safeguards we implement. In a way, it’s great that the “alignment problem” is unsolvable, because “solving” it–and getting people in power to agree to the solution–would lay the groundwork for unprecedented oppression.
