I'm still jetlagged from coming home from CIG 2008 in Perth four days ago (Perth-Singapore-London-Copenhagen is a 26 hours trip from first takeoff to last landing). But it was worth it. As usual, I find the CIG conferences enormously interesting and inspiring. Mostly because it's Computational Intelligence and Games is as much my "core" research area as it gets.
It's true that the acceptance rate it's a bit high, and there are quite a few papers which might not even be good science, in the sense of providing systematic experiments with statistically valid conclusions. In fact, someone who came from the broader CI or ML community and was not specifically interested in the application areas would probably complain about this. (However, it strikes me that most people can easily be made enthusiastic about games when you talk to them a bit... in striking contrast to some other common CI application areas such as scheduling or traffic flow optimization.)
We've beem discussing whether we should lower the acceptance rate, but I don't think we should. At least not yet. The main reason is that the research community is still quite small and we want it to grow. Another reason is that don't really believe in being too harsh. Reviewers can usually tell whether a paper is crap or not, but it's very hard to tell whether it's a seminal contribution. That's for posterity to judge, and to be approximated by the number of citations the paper has amassed after ten years or so. Conferences that only accept 25% or so of papers are, in my opinion, bound to make a rather arbitrary selection. Besides, we now have TCIAIG for publishing the cream of CIG research.
Enough about this, and back to the conference. There were keynotes representing both the ivory-tower variety of CIG research (Jonathan Schaeffer on solving Checkers), the industry perspective (Jason Hutchen, whose keynote I missed due to the conference banquet being the day before. Yes, free drinks) and the harmonious marriage of the two (Penny Sweetser on Emergence in Games).
One paper I particularly liked was Bobby Bryant and Matt Parker on learning to play Quake II using visual inputs only. I think their work is very relevant both for studying the evolutionary emergence of complex intelligence (seeing the FPS as a more advanced robot simulator) and for developing more lifelike NPC behaviour (e.g. aiming behaviour). The paper is not online yet, but here is a previous paper of theirs (NB their results are not very impressive yet, it's the idea I like.)
As for my own contributions, I gave a tutorial (with Georgios Yannakakis) on "Measuring and Optimizing Player Satisfaction". I also presented three papers, one on An Experiment in Automatic Game Design, one on Generating Diverse Opponents with Multiobjective Evolution and one detailing The WCCI 2008 Simulated Car Racing Competition. I hope to find time to write posts on this blog explaining the concepts behing the first two of these papers sometime soon, as I think they are really quite cool myself. I also presented the results of the CIG installment of the ongoing car racing competitions.
This post is already long enough, so I'll stop writing here. What can I say - if you are interested in games and AI/CI, you should have been there! And you should definitely come to the next CIG in Milan, Italy, September 2009. I'll be involved with the organization of that one, so I'll be writing more about it!
Hi Julian,
ReplyDeleteNice wrap-up. I'm actually drafting a similar article but focused more on papers I found particularly interesting. I'll post it to my (brand new and currently contentless) blog in the next day or so (http://harablog.wordpress.com).
Anyway, I thought I'd post because I'm not sure I agree with your suggestion that CIG keep its acceptance rate so high. I think this year was ~75% and, in all honesty, some of the papers presented left me wondering if there was a novel contribution being made at all. I can think of at least one paper which seemed to have no results and another that appeared to solve no problem in particular (I won't mention names; I'm sure you already have some ideas about who the culprits might be).
It is my strong belief that allowing such low-quality work to be published only hinders the reputation of the conference and discourages quality work from being submitted. I realise the community is not very large but growing it shouldn't mean lowering our standards. I know of at least one other community (diagnosis) that is also very small; the main event of their subfield is the DX workshop which, despite its size, is very well regarded.
I also don't see TCIAG as a substitute for a decent peer-review process. CIG shouldn't be a place for draft work (we have workshops for that) but completed pieces of science with results that are reproducible and which make sense (a number of authors seemed to have a poor grasp of their own data which was disappointing). I hope CIG'09 will go some way to address these issues.
On a side note, I also hope the organisers will be more open about who is on the program committee. I was personally disappointed not to know who are the group of people reviewing my work.
Anyway, sorry for the long rant. :)
Hi Daniel,
ReplyDeleteThanks. These are some very relevant points you are raising. As for acceptance rates, it was lower then 75%, but over 50%; I can't remember the exact number now (though I think it was official, but not sure).
You are of course right about the acceptance rate possibly discouraging people to submit. At the same time, why should we hinder people with work-in-progress (or good ideas that they don't have time to finish themselves) from presenting papers in conjunction with the event hosting quality papers? We all benefit from being at the same place and being able to talk to each other.
Maybe we should have some sort of two-tier system. Maybe posters should only be published as abstract. But then, what's the reason for not publishing a full paper, when we don't print "papers" on paper anymore?
This topic is still very much up for discussion, and in the end it is up to the general chair of CIG'09, with the input of all of us.
That the program committee is not listed on the webpage is odd - I hadn't noticed it until now. I'll drop a mail to Luigi and Phil about it.
Hi Julian,
ReplyDeleteI suggest a workshop forum is a more appropriate venue for presenting draft work. A workshop allows for constructive feedback and gives authors an opportunity to take on-board said feedback to improve things.
Without putting too fine a point on it, I believe that publishing poor quality work as part of the main conference proceedings diminishes us all.
Firstly, it becomes harder for outsiders interested in the field to sift the good work from the bad. A direct corollary is that as our reputation diminishes, all papers presented at the conference will be tarred with the same brush: probably bad (regardless of their actual quality).
Secondly, there's no point doing good work if noone will read it and that's exactly the corner we risk painting ourselves into.