They say we should educate people about AI, because we all need to get good at using AI. But what does it mean to be “good at using AI”? I’m not sure. Understanding the technical underpinnings of modern AI models only helps a little bit; I’ve done AI research for 20 years and I’m not sure I’m a particularly skilled user of AI. But here are my two cents, and 2800 words.
It seems to me that there are no magic bullets for efficient AI use. In the recent past there were various incantations you could use that would somewhat mysteriously get you better results, such as telling the model to “think step by step”. Alas, such incantations matter less these days. In general, language models and their associated systems are good at understanding what you tell them, and they improve rapidly.
So what is there to learn? I think the best way to get good at using these beasts is to use them a lot, and try to vary how you use them. I’ve been trying to think of what the main challenges are when using modern LLMs as I’ve interacted with them. Here are some main skills I think you need, in increasing order of technical and existential difficulty.
Expressing yourself clearly
However capable the model is, it doesn’t live inside your head and can’t read your thoughts. You need to tell it what you want from it. You can also not assume that it has the context of everything you’ve experienced in your life. It most likely doesn’t even have the context of the situation you are in right now. Stating what you want clearly is a transferable skill. It is more or less the same skill you need for outsourcing work to a contractor or explaining an assignment to your students. Not everyone is good at it; I have seen many professor colleagues give woefully incomplete or ambiguous specifications to students, for example. Sometimes, I’ve done so myself.
Elucidating your intent via dialogue is useful, but could also lead you astray. It is very useful for the student, contractor, or language model to be able to ask follow-up questions. These may in turn spur you to think of aspects of your original request that you did not think about. You may even understand what you wanted better. However, the follow-up question may also end up leading you in a completely different direction; notice how often an LLM helpfully asks “would you want me to…?”. Expressing what you want clearly from the start is how you actually get the answer you want. And clarity of expression requires clarity of thought.
Appropriate skepticism
Language models are not inherently truthful. At their core, they produce probable tokens. In other words, they produce true-sounding bullshit. In the early days, this meant that you couldn’t really trust anything they said. These days, great strides have been made to reduce confabulations (a.k.a. hallucinations), and if you ask a good language model about something widely known, you can generally trust the answer. In other words, the bullshit is very often true and useful.
A key reason that language models have become more truthful is that they look things up on the web. Basically, they do the same thing as you would: they google things when they don’t know. To understand how important this is, try using a state-of-the-art language model with web search turned off (this is is possible for example with Claude, or if you have a beefy computer that can run good models locally). If web search is turned off and you ask the model about a niche topic that you know well, chances are that it will bullshit worse than a drunk politician.
Now you may wonder, if web search is turned on, do you still need to be skeptical? Yes. Because, as you may have noticed, not everything on the internet is true. And LLMs are gullible.
To understand this better, have an LLM with web search turned on compile a report, complete with sources, for you on a subject you know well. All the leading model providers have “deep research” functions that do this. You will likely find that the referenced material is all over the place: peer-reviewed papers, news articles, forum discussions, even marketing material. It is often hard to know who to trust, and the task is not easier for a language model. It doesn’t matter how advanced the neural network is, it does not magically know things. There is no escaping epistemology. For you, the user, finding out who to trust just got harder, because now the disparate sources are filtered through the same model and presented to you with the same authoritative voice.
A relevant concept here is Gell-Mann amnesia, a concept introduced by Michael Crichton. Yes, the author of Jurassic Park. Gell-Mann amnesia refers to how you forget to doubt statements outside of your area of expertise when they are presented by a source you consider authoritative. Crichton takes the example of reading about the movie industry in a newspaper, and complaining about how the journalists get everything wrong. He would then turn the page to read about something completely different, for example particle physics, and unquestionably accept what he read. But why would the journalists be better at writing about particle physics than they are at writing about the movie industry? Now think back to your experience of asking the LLM something on a topic you know deeply. And then asking the same LLM about something in a topic you don’t know.
Personally, I trust what comes out of a good LLM about as much as I trust what I read in a tabloid newspaper or what I see on TV. Or perhaps as much as I trust a peer-reviewed paper in a venue with loose standards. All of these are useful sources of information, but require skepticism. And exercising appropriate skepticism on a topic you don’t know well is hard.
Knowing what you want
With great power comes the question of what to do with it. LLMs give you great power. At least within certain domains. You know that feeling when open the fridge and just stare at the food inside, not remembering why you went to the fridge in the first place? That’s me, in front of Gemini or Claude, sometimes.
At any given point in time, there’s an infinite number of things you could possibly do. There’s an infinite number of questions you an ask, apps you can build, analyses to run, and so on. Most of them are not what you should be doing right now. In theory, if you always chose the best possible action you could take in order to maximize your overall objective, you would be much more successful than you are right now. But most of the time you don’t think deeply about what to do or ask next, because that would be absolutely exhausting.
Let’s say that you come to your AI tools intentionally, with a concrete task to do. You want to write a text, analyze some data, understand a paper or perhaps create an app. Where do you start? You could simply put in the overall idea into the prompt, something like “help me understand this paper” or “build an app that balances my household budget”. Very likely you will get a result other than what you wanted. This is because any complex request hides a myriad of small design decisions. Either you make those decisions, or the model will make them for you. If the model makes them, it will probably choose very generic alternatives. So you will want to provide lots of details, and likely break down the task in many steps. This, in turn, requires that you actually know what you want to do with the AI system. Not just understanding a paper or building a budget app, but which part of the paper you want to understand and in what terms you want it explained, or which features you want in the budget app and what the interface should be like. Choices choices choices. Making all of those choices is hard work, but it’s your work.
Knowing the other
There is a tendency to look at what an AI model does best, and think that that is how “intelligent” it is. But your view of intelligence is always relative to some implicit idea you have of what a human can and can’t do. But that is not how AI works. Whatever an LLM is, it is not a human, and does not have a human-like distribution of skills.
The very same LLM that knows more than any human has ever known and writes working software from scratch can entirely lack spatial intuition, make ridiculous errors in image generation and have the memory of a goldfish, forgetting the start of your conversation. It is very confusing, because your intuitive notion of intelligence keeps intruding and insisting that if someone is good at A, they should also be good at B, like a human would be. The sensible thing to do is to forget, or at least put aside, your notion of intelligence and start keeping track of capabilities to do particular tasks. Which AI model is best at translating to your native language, and how good is it? Which one is best at synthesizing data from the web, or writing frontend code? And so on.
To make matters worse, new and better models are released all the time, and the same model often comes in different sizes with different capabilities. The best you can do is to use AI often, use different models, and for different tasks, to get a good idea of what the models can do. And, again, to abandon the concept of intelligence. It is not useful here.
Knowing yourself
Eventually, you have to confront who you really are. Or at least what you are good at and what you want to get better at. This also means choosing which skills you can afford to let atrophy. Everything you do together with an AI system is a collaborative work to some extent, and you need to choose which parts you want to do yourself. You only have so many hours in the day, and much fewer of them that you can truly focus. Where do you want to spent your limited cognitive resources? For example, do you want to write a text yourself and have the LLM critique it, or do you want to let the LLM write it based on an outline you’ve written? Both paths are possible, but give different results in terms of style and, presumably, quality. One of these paths is much more work than the other. But that same path also results in a text that which is written in your own style, a deeper understanding of what you wrote about, and an opportunity to develop your skills as a writer. Is it worth it to write the first version of the text yourself?
One way of answering that question is to do the work yourself where you provide the most value. It’s a matter of what your comparative advantage is. Is your time better spent writing this text, or doing some other part of the complex work that you are trying to do together with an AI system? But this is tangled up with the question of which of your skills you are proud of, and what you enjoy doing. Maybe you think of yourself as an idea person, but you really enjoy editing text, and you are better at drafting a first version of the text than you are at either coming up with the ideas or doing the edits. The LLM can theoretically do all of these things, but then it’s not your work. If you only have time to do one of them, which one do you choose? Only you can answer this question.
The problem is further complicated by the fact that the way you get good at things is by doing them, and the best way to lose a skill is to not practice it. Handing over your tasks to the AI system means that you lose a chance to get better at doing those tasks. A little bit like how you don’t get any exercise if you drive to work instead of walking, but it does get you there faster.
When you build something complicated, there is also the issue that you only really know how something works if you built it yourself. This is a common pitfall when using AI to build software for you. Initially, you make great progress by “vibe coding”, and it is oh so satisfying to see all that code scrolling by as it is written in response to your requests. You just tell the AI system that you want some functionality, and mere seconds later it is there! However, at some point you run into problems. Some part of your program is not working like it should, and you don’t know why. The LLM doesn’t seem to know either. So you decide to go into the code base yourself–after all, you know how to write code–but you don’t understand it, because you’ve never seen most of it. In extreme cases, you may resort to rewriting it from scratch, so you actually know what’s going on.
What Socrates didn’t know
Expressing yourself clearly, appropriate skepticism, knowing what you want, knowing the other, and knowing yourself. Is this what you need to use AI well? Perhaps, but if so, Socrates would arguably be a master AI user. This seems like an outrageous idea, that could only be dreamt up by someone who was a philosophy student before he became an AI researcher and sometimes wonder whether he should have stuck with philosophy (me).
But let’s take it seriously. Would Socrates be a master prompt whisperer? Maybe. There’s certainly something appealing about Socrates using his eponymous method to coax unknown truths about the world out of unsuspecting language models. And it would be incredibly interesting to see what came out of such an experiment. (Maybe the models have managed to come up with some profound truths in their quest to abstract all the text we have fed them?)
However, I don’t think Socrates would be very effective at using AI in the world we actually live in. Why? Well, because he lived 2500 years ago in a slave-holding iron age society. Socrates famously stated that he knew only that he knew nothing. By modern day standards, he was right. He knew nothing about e.g. finance, software, logistics, aerodynamics, marketing, corporate law, municipal bureaucracy, TikTok, and all the myriad other things we do for fun and profit in the modern world.
And here’s the rub: to express yourself clearly, exercise appropriate skepticism, and know what you want you must know the domain you’re working within. If you don’t, you are not likely produce anything very valuable, with or without AI.
Some people see the huge and growing capabilities of modern AI as a sign of that human knowledge will be less important, perhaps even unimportant, in the future. Why know things, when you can just drink ask the AI to do things for you? The AI knows best, right? But you don’t know what to ask for if you don’t know things. The amount of things that could possibly do at any given time is practically infinite, a fact that is hard to wrap your head around. It is, in general, impossible to know what the optimal thing to do is, even if you know what you want to do. AI systems add agency to us in much the same way as all the other machinery of civilization, from cars to corporations, from light bulbs to libraries. It is more important to know things now than it was in Socrates time, because there are so many more possibilities. I think that knowledge will be even more important in the future.
I think this is true even if the AI system you use knows more than you about whatever you want to do. For example, assume you want to analyze some data. Unless you have a degree in statistics, modern frontier AI models probably know more about statistics than you do. Still, the more you know about statistics, the better you can specify what kind of analysis you want the system to do on your data. You are also more aware of what information can be reliably gotten at all. You will probably also understand the results of the analysis better, and able to refine the analysis better. Crucially, you will also be better prepared to point out when the result is wrong. The more you know, the better, but even just understanding the difference between mean and median helps. Yes, there are people who don’t. Yes, adults.
I think the same argument goes for essentially every other domain you can imagine an AI system helping you with, from fiction writing to airship design to proving mathematical theorems to understanding stale memes. So, go out there and learn things. And use AI a bunch. And think critically.