We’d like to take this time to respond to the heated debate that followed from our “What’s Wrong with the IGF” post.
First, to anyone that took our blog post personally, please understand that this was not a personal attack.
We wanted to bring to light a fact that some judges don’t play games that they are assigned, and that that may be a problem in the future.
In light of the $95 entrance fee, we believe every developer deserves a fair shot at a nomination.
Judges not playing a game they are assigned to judge, for any number of minutes, is simply not acceptable.
Regardless, we had not intended this as a personal attack against Brandon Boyer or Simon Carless.
We understand that they do their best to make the IGF what it is.
As hard as they work, the system itself is flawed because it makes it easy to overlook games when the IGF was created to do the opposite: notice overlooked games.
However, we do not agree that calling would have solved anything. If we had decided to call, it would have been a word of honor against posting.
Furthermore, we mentioned the email because we are arguing for openness, not private correspondence.
Everyone should know about these issues and not be kept in the dark, so that we can have an honest debate over what could be done.
It is enlightening that some developers have already come out of the woodwork to voice similar complaints.
To that end, we’d like to wish everyone the best. Oh, and we’ll change the blog theme.
- the rotting cartridge
To anyone questioning the data, it is posted below (numbers are in minutes). Please note that out of respect for the judges, names are stripped from these numbers, as they should always be for anyone making a similar post.
Judges:
5.6,13.4,0,0,0,2.5,13.1,53.2
Beta Testers:
12.5,28.6,6.8,34.7,174.3,80.1,1.8,34.5,39.9,75.3,4.3,4.6,4.1,1.0,25.4
[...] Posted in Cartridges, Features, Game Development, Gaming, Uncategorized | 62 Comments The Rotting Cartridge Developer Interview: Design RE: What’s Wrong with the IGF [...]
I agree. If they’re charging a fee and are going to act as the face for indie game development, they need to prove that they’re providing a fair and accurate judging process. If judges can’t be held accountable for fair voting, another organization that is willing to do so needs to step up and replace the IGF.
I guess you could say I had a similar experience, although I never tried contacting the judges or IGF itself about it since I thought it was standard practice
Judges:
4.5, 1.1, 22.1, 0, 0, 0, 0
As for beta testers, the game has had probably 10 or so hours logged by around 15 people, but there are quite a few builds so it’s not really an objective comparison either way.
The theme was fine. Blank is not an improvement. Oh well, you can’t please everybody.
I dunno, maybe you should’ve taken up their offer for a private discussion and gone public only if you weren’t satisfied with the resolution. This just seems like a dick move to go public without having first tried to resolve things amicably in private, a desire for “openness” notwithstanding. Y’know, you could’ve expressed your desire for openness without having gone public with this in the first place. Openness can happen without you having to turn it into an expose. That just creates this antagonistic and divisive mess that *you* have instigated. In fact, I’d go so far to say that you’ve really hurt your chances at “improving openness”. I’m sure that next time around, no one will have any idea what the judges did with their games.
In conclusion, I think all you’ve really demonstrated is that you really want attention. Well, you got mine. I’ll be sure to avoid your games in the future. Have a nice day.
If you could stop dick-riding the IGF staff for 10 seconds, that’d be great. And as much as you’d like every issue to be hushed up and ignored behind closed doors, No, go straight to hell. The only antagonistic and divisive mess here is the IGF and their defenders.
Wow. Your reply is so crazy. If you were a developer who submitted work, you’d feel entitled to this information. Openness begets openness. Next time around people are going to ask lots of pointed questions.
Your stance is that judges must, with 100% success rate, judge all of their assigned games? I disagree.
A game must 100% be judged, by multiple judges, but this already happens.
Multiple judges are assigned to games for a reason. In fact, games are over-assigned to judges to account for missing scores (compatibility issues, some personal/work deadline slams a judge and they get busy, etc).
I wrote the IGF judging backend that’s still being used. There are a LOT of admin-only scripts to provide visibility into which games have a low amount of scores. I would much, much rather that judges *not* score their assignments if they get busy. When games have an unusually low amount of scores, somebody looks into the problem. This might mean emailing the entrant (broken build/upload), or simply assigning more judges to the game.
The alternative is one where judges phone in their scores to meet their assignments. As the database guy, I have NO way of detecting these scores. I vastly prefer clean data, which is to say some percent of judges flake out and are easily accounted for.
To my knowledge, no game has EVER received zero judge scores without approval from the entrant (including some cases where extraordinary measures were taken to get a working build or play session).
Your stance contradicts other developers’ experiences that have been brought to light due to this issue. We are striving for perfection, but with the understanding that we will not reach it. That is better, in my opinion, than Mr. Boyer’s stance that “not all judges need to play their assigned games.” Let’s take a look at some numbers:
1 game has 8 judges assigned to it. Now on average, let’s say 4 judges play each game, which is okay by Mr. Boyer’s standards, as he outlined in his response to our original post. Each of these judges has a 1/2 chance to play the game. However, the distribution is random. This means that there is a probability that fewer than 4 judges will play any particular entry. This can be approximated by a binomial distribution (http://www.wolframalpha.com/input/?i=x+%3C+4+binomial+distribution+n%3D8+p%3D0.5) with n=8, p=0.5, meaning the percentage of games with fewer than 4 judges that played the game is 36.32%. For 570 games, 207 games got played by fewer than 4 judges. This is what we are referring to as “not acceptable.”
Even if the probability is 5/8 judges per game, which is what happened in our case, that’s still 78 games that got less than 3 plays.
This isn’t something that should be “okay.” Those are 78 developers whose games weren’t given a fair shot, by Mr. Boyer’s own standards. If we count one judge per game dropping out due to extenuating circumstances, and all of the other judges playing the game for any number of minutes, the result is < 1 game where less than 4 judges played. If Mr. Boyer's argument is true and a game only needs at least 4 judges to determine a nomination, then the ideal number to satisfy his argument would be that 7/8 judges, on average, played each game. As you can see it does not need to be the perfect number 8/8.
If the IGF is willing to release their own statistics and show that we were one of the extremely unfortunate approximately 38 developers that got played by less than 6 judges, then our story is rare (http://www.wolframalpha.com/input/?i=x+%3C+6+binomial+distribution+n%3D8+p%3D7%2F8). But I suspect that this is not the case, and that the average number of judges that played each game is not 7, but much lower.
(If anyone has any problems with how I came up with these numbers, please tell! Keep in mind the ’8 judges per game’ figure is speculation, because we do not really know how many judges are assigned to their games, this is just our experience.)
Except: 8 is on the low side for assigned judges (because it was iOS). The game assignment script runs through platforms in order of rarity, in order to make sure judges with dev kits, iOS, etc hardware get games first.
This means that “rare” judges fill up, so platform-specific games generally have fewer judges.
I appreciate the mathematical effort, but it’s based on some assumptions that you don’t even realize *are* assumptions (that you are the average, that judge counts are fixed, etc).
That aside, the solution is still to get more judges. In general–not just IGF!–there are two options when it comes to management:
1) Relying on measurable data, optimize the system towards your desired goal
2) Ignoring real-world circumstances, impose unrealistic goals on individuals
You’re leaning towards #2 by asking for all judges to fulfill all judging assignments all the time. Don’t take this personally, but I hope you aren’t in a real-life managerial position. Option 2 makes for terrible, terrible managers :/
@MatthewWegner – It’s “unrealistic” to expect a volunteer judge, who is assigned 20 games, to spend a total of 5-10 hours over the course of a few months? Because 20 games at 15 minutes each is 5 hours, and at 30 minutes each is 10 hours … You need to drastically re-evaluate the kind of people you are asking to judge for you.
Yes. It is unrealistic to ask judges (in July), if they’ll be able to judge games over a ~6 week period (in November/December), and expect that 100% of judges will judge 100% of their games. Shit happens, people get busy, games don’t work, and on and on and on.
100% is a stupid goal. I don’t know why people are so fixated on it, especially because it’s so clear the system is designed to operate with judge dropout.
To point to <100% of judging completion rate as an issue is so terribly misguided I feel like a broken record by continuing to point that out. Focus on judge quality and depth of experience. 5 minutes of play time is so much more of a red flag than 50% completion that I'm honestly surprised it's been the focus of this post and others.