Friday, October 10, 2014

Why academics and game industry don't collaborate on AI, and how we could improve the situation

In academia, there is a research community focused on artificial intelligence (and computational intelligence, which is mostly the same thing) in games. Within the games industry, there are job titles such as AI programmer and game components known as AI, and there is also a community of people self-identifying as game AI experts. Each community has their own conferences, publications and associations. The CIG and AIIDE conferences, the TCIAIG journal and the Games Technical Committee on the academic side, and the GDC AI Summit, Game AI Conference, AIGameDev website and AI Game Programmers' Guild on the industry side.However, there is surprisingly little collaboration between these communities. Instead, there is a large degree of mutual ignorance and distrust. There are a number of shining examples of crossover of industry knowledge into academia or vice versa (these are the subject of a different blog post) but these are unfortunately exceptions to the rule. Every now and then, someone comes along and points out the lack of collaboration, and vigorous discussion ensues. Every time this discussion starts, many people who have not thought about this situation are honestly surprised (they might not even have known that there was game AI research in academia or game AI R&D in industry). I've seen this debate playing out on repeat dozens of times by now, so I thought I'd share my thoughts about it in a compressed form for future reference.First, here is an incomplete list of reasons why academics and game developers that work on AI don't collaborate, and often don't even understand each other.
  • Commercial game AI and academic game AI only partly overlap. There are problems that are studied in both communities, but also technical problems in games that are considered too uninteresting to be AI by academics and academic problems that are too abstract or weird to be interesting to industry along with academic methods that won't work in real games.
  • Some (many?) academics are not really interested in doing work which is useful in games, just in using games as testbeds.
  • Some academics want to do work that is useful in games but do not know the demands and constraints of game design and development.
  • Some academics want to do work that is useful in games but do not have access to code and/or data to do that. Industry generally won't make code and data, or even APIs, available because of the need to protect intellectual property.
  • Some academics want to do work that is useful in games, but target the wrong type of games. In particular, there is the perception that only AAA games in well-established genres like FPS, RPG and RTS are worth targeting, while these games are often designed so as not to need much (interesting) AI. Why work on solving AI problems in a game which was designed not to need AI? Instead, the lust and need for new AI techniques is often stronger in the indie scene and emerging game genres.
  • Some developers in industry harbour a dislike or distrust of academic research in general. This could be because the game industry tends to see itself as an entertainment industry rather than a tech industry. It could also be because certain developers were very dissatisfied with their own university educations.
  • Some (hopefully very few) academics have a superiority complex and think their fancy degrees entitle them to talk down to the unwashed masses.
  • Some (hopefully very few) developers in industry have a superiority complex and think the fact that their software sells entitle them to tell others how Things Really Are.
  • There is little or no funding available for game-applicable AI research from government funding agencies, because games are not seen as important enough when you could instead do research on curing cancer or stopping terrorists.
  • There is no funding available for game-applicable AI research from companies, because game development companies don't support research, unlike companies in many other sectors. This could be because of the very short time horizon of game development companies. A car company or insurance company will have a 20-year strategic plan, a game company most likely won't.
  • The academic incentive system - publish, get cited, get grants - does not reward doing the "last mile" of converting your research prototype into something that could be plugged into a game engine.
  • The game industry's incentive system - make sure the next game sells or you're out of a job - does not reward trying things that might not work.
  • We simply don't go to the same places, because GDC is too expensive for academics and does not have any publication options, and academic conferences are deemed irrelevant by industry.
Perhaps most of these reasons point to an underlying difference in motivation. Researchers want to do crazy long-term stuff, developers want to do safe short-term stuff. I'm certainly of the crazy persuasion myself. But this shouldn't stop us from collaborating - rather, this complementarity is a fantastic opportunity. (Indeed, some of my own best research-work has been in collaboration with a more down-to-earth personality who tempers my wild flights of fancy.) We just need to find ways of collaborating, and stop blaming each other.I'm convinced the intersection of the interests of academic researchers and industrial developers is large enough to permit much more collaboration than we see right now. So how could we improve this situation? Here's some suggestions, again an incomplete list:
  • Respect each other. Don't slander each other.
  • Be curious about each other's work, both in terms of contribution and motivation.
  • Write blog posts (not just academic papers written in hard-to-penetrate academese) and post videos explaining your work. Goes for both academics and developers, maybe particularly for academics.
  • Find ways of making datasets and source code from commercial games available, perhaps with an NDA; alternatively ship games with an API to interface AI.
  • Make it easier to publish academic work based on collaboration with game developers. Special sessions of conferences and special issues in journals.
  • Reward developers for going to academic conferences.
  • Make it possible to publish academic papers at industry conferences, alternatively just make it cheaper to attend for academics.
  • Place more value on published (or otherwise publicly released) games and playable experiences in academics' CVs.
  • Fund applied research into game AI.
  • (Ok, I'll stop joking. Not very likely that this will happen. We'll make this work without funding because we love games and AI.)
It would also be worthwhile to point out that there are more than one way for an academic to make an impact on games. Some different modes of working - and I don't have to remind you this is an incomplete list and the different modes of working are certainly not mutually exclusive - are:
  • Read up on what industry game AI developers are interested in and which problem they find hard to tackle. This can be hard, as the information is typically not available in the form of academic papers. But AIGameDev.com is a good starting point, so is the AI Game Programming Wisdom book series. Stalking talkative developers on Twitter might also help.
  • Approach game developers and ask if you can work together with them.
  • Send your students on internships in game companies, with the objective of doing some project both you and the company find interesting. This has worked well for me in the past.
  • Use benchmarks that are based on industrially relevant games. Several of the competitions that run each year at CIG and AIIDE - in particular the BotPrize and the StarCraft competition, but to some extent also Pac-Man and Mario AI (especially if you're not just focusing on performance but on other aspects of gameplay or content generation) - are relevant here.
  • Make your own games. You don't have to work with the game industry in order to make an impact on games. In fact, you don't even have to work on the problems the industry define for you. If you are the kind of researcher who likes to come up with crazy new algorithms and then invent the problem they might solve, go one step further and build a whole game around it. Or many games. Release them out in the wild, let people play them even if they are not perfect (they will not be). They serve to prove a point, that this is AI which is relevant for games, even if it's not what the industry calls "Game AI". Lead, don't follow. Redefine what games are, and what AI is. Who's stopping you? I think this kind of research is the one which is most likely to end up being really important.
  • Participate in game jams. This is an excellent way to get started with getting your hands dirty and actually making games. At least, that's what I hear, I never have time to participate in game jams because I'm busy arguing with people on the Internet.
In short, there are so many cool things to do that we should all stop complaining about our various differences and just get on with doing great work. So what are you waiting for? What am I waiting for? I'll stop writing now.
Acknowledgements: this mail started as a reply to a mail by Robert Zubek, which was a reaction to an ongoing discussion in the AI Game Programmer Guild's mailing list, which in turn partly continues the Twitter debate following Squirrel Eiserloh's recent keynote at AIIDE. My views have been shaped by ongoing discussions about this topic with more people than I can remember in various mailing lists and conferences. People whose opinions on this topic have illuminated my own recently include Dave Mark, Alex Champandard, Tommy Thompson, Mike Cook and Luke Dickens.

2 comments:

Richard C. said...

Very good article (IMHO). I would add another important reason which in addition to the mentioned ones prevents industry and academia from more cooperation: Game AI researchers are often interested in the "grand questions" which point towards general solutions for arbitrary games - or at least on a more abstract or general solution for a particular problem recurring in many games. An extreme example would be the research area of general game AI with competitions like the "General Game Playing Competition" or the "Arcade Learning Environment". General game AI of course is important fundamental research, but it is questionable if game industry might benefit from it in the next years. Rather it is obvious that within such a hard market there are no financial resources left for such fundamental research. A game AI being able to play Chess and Go and further, similar (maybe even unknown) games, would very probably not be even nearby a chance winning against the strongest Chess engine or Go engine - at least for now and the next 10 years. Regarding a single task, a special solution often outperforms the general one, as a hybrid vehicle which can fly as well as drive on a road ("flying car") will always be outperformed by a one-seated plane or a simple car (regarding the task to accomplish).

When it comes to less deliberative but more reactive AI (like enemies in an FPS game), a general AI player, e.g. adapting to the game by RL (which theoretically could tend to an optimum) in many cases might again be more expensive than a near-optimal hand-coded solution (since in RL so far there is still much to be done regarding state space (feature) engineering, tuning etc.).

Game AI developers in industry are interested in the most competitive and affordable solution for a single game while aspects of reusability or abstraction are mainly covered from the viewpoint of clean software engineering (OOP, design patterns etc.). These guys know how to implement MCTS with UCB, and maybe so far it's all they need from the researchers - a paper to read from time to time... these papers are available for free anyway, so I guess they read them (thus there is maybe more "onedirectional cooperation" than we think).

Kind regards
Richard C.

Nonadecimal Creative said...

As a game developer, I'm always trying to learn from academic research and apply it to my own projects. Establishing partnerships with some of them to expand the research into focused game concepts would be wonderful.