Monday, April 18, 2016

The differences between tinkering and research

Some of us have academic degrees and fancy university jobs, and publish peer-reviewed papers in prestigious journals. Let's call these people researchers. Some (many) others publish bots, hacks, experimental games or apps on blogs, web pages or Twitter while having day jobs that have little to do with their digital creative endeavors. Let's call those people tinkerers.

So what's the difference between researchers and tinkerers?

This is a valid question to ask, given that there are quite a few things that can be - and are - done by both researchers and tinkerers. Like creating deep neural nets for visual style transfer, creating funny Twitter bots, inventing languages for game description and generation, writing interactive fiction or developing Mario-playing AIs. These things have been done by people with PhDs and university affiliations, and they have been done by people who do this for a hobby. Anyone can download the latest deep learning toolkit, game engine or interactive fiction library and get cracking. So why is this called research when the academic does it, and just a curious thing on the Internet when a non-academic does it?

Let me start by outlining some factors that are not defining the difference between tinkering and research.

First of all, it's not about whether you work at a university and have a PhD. People can do excellent research without a PhD, and certainly not everything that a PhD-holder does deserves to called research.

Second, it's not because research always is more principled or has a body of theory supporting it. Nor is there typically a mathematical proof that the software will work or something like that. It's true that there are areas of computer science (and some other disciplines) where research progresses trough painstakingly proving theorems building on other theorems, and I have a lot of respect for such undertakings, but this has little to do with the more applied AI and computer science research I and most of my colleagues do. On a technical level, much of what we do is really not very different from tinkering. Some of it is good code, some of it bad, but typically there is a good (or at least interesting) idea in there.

Third, it's not really the publication venue either. It's true that most of us would trust a peer-reviewed paper in a good conference or journal more than something we find on a blog, and peer review has an important role to fulfill here. But what was once a sharp boundary is now a diffuse continuum, with a myriad of publication venues with different focus and different degrees of stringency. Quite a few papers make it through peer review even though they really shouldn't. Also, the traditional publication process is agonizingly slow, and many of us might just put something online right away instead of waiting until next year when the paper comes out. (I personally think it's prudent to always publish a peer-reviewed paper on substantial projects/artifacts I contribute to, but I sometimes put the thing itself online first.) It is also becoming more common to post preprints of papers on places such as arXiv as soon as they are done, and update them when/if the paper gets accepted into a journal or conference.

So we are back to square one. What, then, is the actual difference between tinkering and research? Let me list four differences, in order of decreasing importance: scholarship, testing, goals and persistence.


Probably the most importance difference between tinkering and research is scholarship. Researchers go out and find out about what other people have done, and then they build on that so they don't have to reinvent the wheel. Or if they do reinvent the wheel, they explain why they have to reinvent the wheel and how and why their wheel is different from all the other wheels out there. In other words, researchers put the thing they have done into context.

For example, almost seven years ago I made some experiments with evolving neural networks to play Super Mario Bros, and published a paper on this. The work (and paper) became fairly well-known in a smallish community, and a bunch of people built on this work in their own research (many of them managed to get better results than I did). Last year, some guy made an experiment with evolving neural networks for Super Mario Bros and made a YouTube video out of it. The video certainly reached more people on the internet than my work did; it makes no mention of any previous work. Seen as tinkering, that work and video is good work; seen as research, it is atrocious because of the complete lack of scholarship. The guy didn't even know he was reinventing the wheel, and didn't care to look it up. Which is fine, as it was probably not meant as research in the first place, and not sold as such.

Good scholarship is hard. It is easy to miss that someone tackled the same problem as you (or had the same idea as you) last year, or 5 years ago, or 50. People use different words to describe the same things, and publish in out-of-the-way places. Once you found the literature you must read it and actually understand it in order to see how it is similar to or differs from the idea you had. Because good scholarship is not just listing a number of previous works that are vaguely related to what you, but rather telling a believable, coherent and true story where all those previous works fits in, and your own work makes a logical conclusion. Therefore good scholarship takes a lot of searching and reading, a lot of time and effort. It's no wonder that lots of people don't want to spend the time and effort, and would rather get on with the tinkering.

It's common for incoming PhD students and other students to question the need for the scholarship when they could spend their time writing code. So let me go through some reasons for doing good scholarship, in order increasing importance. (Beyond wanting to get a PhD, of course.)

The perhaps most immediately important reason is common civility and courtesy. If you do a thing and tell the world about it, but "forget" to tell the world that person X did something very similar before you did it, then you are being rude to person X. You are insulting person X by not acknowledging the work she or he did. Academics are very sensitive to this, as proper attribution is their bread and butter. In fact, they will generally get offended even if you fail to cite other people than themselves. Therefore, the easiest way to get your papers rejected is to not do your related works section.

What about someone who doesn't care what academics think, or about getting published in peer-reviewed journals and conferences? Any point in spending all that time in front of Google Scholar and reading all that technical text written by academics with widely varying writing skills? Yes, obviously. Knowing what others have done, you can build on their work. Stand on the shoulders of giants, or at least atop a gang of midgets who have painstakingly crawled up on the shoulders of taller-than-average people. The more you know, the better your tinkering.

But to see the primary reason to do our scholarship before (or during, or after) tinkering we must lift our eyes beyond our own little fragile egos (yours and mine). It is about the accumulation of knowledge and progress on the scale of the human species. If we learn from each other, we can ultimately push the boundaries of what we collectively know forward and outward; if we don't learn from each other, we are bound to do the same things over and over. And it's way more likely that others will learn from you if you make it clear how what you are doing is different from (and maybe better than) what was done before.

So if you want your little hack or bot or whatnot to contribute to science, in other words to the evolution of humanity, you should do your scholarship.


Here's another big thing. A tinkerer makes a thing and puts it out there. A researcher also tests the thing in some way, and writes up what happens. Tests can take many shapes, as there are many things that can be tested - it depends on what you want to test. Generally the test is about characterizing the thing you made in some way. It could be performance on some sort of benchmark. Or a quantitative characterization with statistics from running your thing multiple times. Or maybe a user study. Or why not a qualitative study, where you really take your time to interact with your software and describe it in detail. The point is that if something is worth making, it's also worth studying and describing. If you don't study it when you're done, you're not learning as much as you could. And if you don't describe it well, nobody else will learn from it.

Interestingly, the tinkering and testing can sometimes be done by different people. There are quite a few academic papers out there that systematically study software that other people built but did not care to study in detail. This ranges from performance analysis of someone else's sorting algorithm, to large parts of the academic field of game studies.


Why do you tinker? Because of the thrill of trying something new? To hone your skills with some tool or programming language? To build useful tools for yourself or others? To get attention? To annoy people? Because you had an idea one night when you couldn't sleep? All of these are perfectly valid reasons, and I must confess to having had all those motivations at point or another.

However, if you read a scientific paper those are usually not the stated reasons for embarking on the research work presented in the paper. Usually, the work is said to be motivated by some scientific problem (e.g. optimizing real-value vectors in high-dimensional spaces, identifying faces in a crowd, generating fun game levels for Super Mario Bros). And that is often the truth, or at least part of the truth, from a certain angle.

While tinkering can be (and often is) done for the hell of it, research is meant to have some kind of goal. Now, it is not always the case that the goal was to get the result that was eventually reported. A key characteristic of research is that we don't really know what the results will be (which is why most grant applications are lies). Sometimes the result comes first, and the goal afterwards. Fleming did not set out to discover Penicillin, but once he did it was very easy to describe his research as solving an urgent problem. Also, he had been working on antibacterial compounds for a long time following different leads, so he recognized the importance of his discovery quickly.

Usually, goals in research are not just goals, but ambitious goals. The reason we don't know what the results of a research project will be is that the project is ambitious; no-one (as far we know) has attempted what we do before so our best guesses at what will happen are just that: guesses. If we understand the system so well that we can predict the results with high accuracy, chances are we are tinkering. Or maybe doing engineering.

Of the papers I've written, I think most of them started with some kind of problem I wanted to solve, so in other words a goal. But many others have been more opportunistic; we had a technology and an idea, and wanted to see what happened because... well, it sounded like a cool thing to do. Interestingly, I have never found it a problem to describe the research as if we had a particular goal in mind when we did it. This is probably because I always keep a number of high-level goals in mind, which implicitly or explicitly help me shape my research ides. This brings us to the next difference between research and tinkering.


You know Einstein's paper that established the special theory of relativity? A guy in his twenties, having published one a few papers before, publishing a single paper that revolutionized physics? Most papers are not like that.

Most papers report tiny steps towards grand goals. Results that are not in themselves very exciting, but hopefully will help us sometime in the future solve some problem which would be very exciting to solve. Like generating good video games from scratch, curing cancer or algorithms that understand natural language. The vast majority of such breakthroughs don't just happen - they are the results of sustained efforts over years or decades.  Recent progress we have seen in Go-playing builds on decades of research, even though it is sometimes reported as a sudden move by DeepMind.

Tinkerers are content to release something and then forget about it. Researchers carry out sustained efforts over a long time, where individual experiments and papers are part of the puzzle.

Doing research therefore requires having goals on different time scales in mind at any given time, and being able to extract high-level goals from level-goals, and seeing where new results fit into the bigger scheme of things. That is why I consider my attempts (together with colleagues) to chart out the research field and establish grand challenges as some of my most important. See, for example, our paper on challenges for procedural content generation, or on challenges for neuroevolution in games, or our attempt to structure all of research on AI in games.

Interestingly, when I started doing research I did not think I had much persistence at all. I also did not understand how much it was needed. Both of these realizations came later.


This post was inspired by my recent reading of Matti Tedre's "The Science of Computing", a history of the debates about what computer science actually is. He argues that computer science has variously been seen as mathematics, engineering and science, and that this debate has gone back and forth for as long as there has been computing researchers, with no clear end in sight. Reading the book, I felt that most of my research is not science, barely engineering and absolutely not mathematics. But I still think I do valuable and interesting research, so I set out to explain what I am doing.

The post was also inspired by discussions and arguments with a number of my colleagues, some of which have rather exclusionary ideas of what Science is and how Truth should be attained; and others who don't seem to think there's much difference between a blog post and a journal paper and who question the need to do all of the boring parts of research. I hope I have been able to make a good case for doing good research.


Unknown said...

Sir Alexander Fleming discovers benzylpenicillin, Ian Fleming discovers James Bond. Basic research starts with Wikipedia?

Julian Togelius said...

Fixed. Not a question about research as much as writing speed.

SHAH said...

Quite a thrilling post! Enjoyed reading...I am myself a tinkerer, doing on and off research on Quantum Computing And AI. Thx for the wonderful article. Stay blessed

Unknown said...
This comment has been removed by the author.
Hemal said...

Interesting article, however there is one thing missing. The tinkering guy many times does not have access to those fancy papers. Don't get me wrong but this is a fact that people at elite universities have all the subscriptions and people working on their own have less or no access to them. Buying papers is not a gem of solution for the tinkering guy because every topics has tons and tons of peer reviewed papers now. Moreover, I don't know of other fields but computer science guys do not regularly put the code for using their method. The guy who just wants to try the method and use it to build something new has to redo the work anyways because there is no code and the language of the paper is at times very "scientific" and make papers hardly reproducible.As much as I agree with many points that you put forward I think that the tinkering guy would use or reuse. You made some good points there Kudos :-)

Unknown said...

One thing perhaps of note is that SethBling, i.e. the youtube mario guy, may have actually contributed something new when he applied the same algorithm to MarioKart (see here ) which is, as far as I know, a novel learner representation for racing games. Indeed, it's the only instance that I have encountered of the local perception grid style of representation being used for a 3D game. Of course it's entirely possible that I'm just showing my lack of due diligence and that the same thing has been done a thousand times over.

Either way (for if I have been lazy, it has at least introduced me to the idea) it goes to show that there is value to be gained from tinkerers (not that I believe that you were suggesting otherwise). Furthermore, I believe the suggestion was given to him by the community, so his latter experiment may never have happened if the idea had remained buried in some paper somewhere.

Al said...

@hermal: in biology you would have a good point, but in computer science (especially AI / machine learning) the papers are open to the public and not hidden behind paywalls.

Danny said...

Looks like you didn't care to watch the video. From 4:40 the guy makes plenty of reference to prior work, names authors of papers and calls out the name of the methodology used.

Your statement "The guy didn't even know he was reinventing the wheel, and didn't care to look it up." is completely and blatantly false, making your criticism very weak.

Julian Togelius said...

@Danny: He mentions NEAT, the algorithm he uses, and that's good. But that's the only work he mentions. He does not mention any approaches to playing Mario, or even solving a similar problem. A real discussion of previous work would look at different approaches to this type of problem, and their pros and cons. And if there's someone who has done essentially the same thing before, that should be mentioned...

Note that I'm not even criticizing SethBling here. He did a great job with that video, and there's no law that you need to have a related works section in your YouTube videos. Having it would probably have made the video less engaging. I'm just using him as an example of when something is not research, because of the lack of scholarship.

Andrew said...

By calling him "Some guy" repeatedly and never using his username or given name, it very much comes off as criticism and it did to me as well. Thank you for clarifying in the comments that it isn't that, but I got exactly the same feeling from this that Danny did due to diction.

Besides that, the article is helpful and interesting. Thank you.