Digifesto

I’m inspired by Mel Chua’s recent posts about culture shock of entering academia from the open source. I don’t have her humility about it and so am convinced that they are doing a lot of things wrong more or less from the get-go, so I’m more or less looking for problems. That said, one came up at lunch with an old friend who’s finishing up his PhD.

My friend Joe reports that sometimes, when papers are submitted to conferences, academic holy war disputes will sometimes affect whether papers get accepted.

Ok, maybe that doesn’t sound like much of a surprise, but it’s an interesting mechanism.

According to Joe, conference papers are reviewed by attendees. Informally, somebody who gets their paper accepted is required to review 3 or so other papers. Nothing bad there.

However, when there is a “holy war” — a major division within the field about a basic theoretical or methodological issue — these religious persuasions will affect the reviews and lead to some papers being rejected despite what we could suppose to be their objective merits.

Is this bad? Is it any different from the open source process? I think so.

But not because of the dispute itself. There’s got to be some substance to these kinds of theoretical and methodological differences. Gosh, open source is full of divisive holy wars, and in general they are a good thing, since competing camps race to innovate and prove that Python is a better programming language than Ruby, or whatever.

The difference in the academic domain is that these conferences are a bottleneck for publication and accreditation, and the conference process itself is not easily forked. So the paper selection process is not merely curational, in the sense of selecting papers of interest for the attendees. Rather, rejected papers are silenced and discredited.

Some balance has to be struck. There has to be some venue for a real conflict of ideas, because unless you fight the holy war, how can you find out who is right? On the other hand, since individual’s reputations are tied to the success of their religion, there is an incentive for doctrinaire and skulduggerous rejection of opposing papers without regard to how much these papers contribute to the field.

What’s the solution? We could imagine a more open, web-based unconference system for accepting papers. There could be the same requirement that one has to review other papers in order to get ones own paper included. Reviews can include rating metadata that affects its prominence within the conference; reviewers could also be rated (for their comments) to give them additional clout within the community. Then you could track discrepancies in people’s ratings on controversial items to detect where the holy wars are at and correct for them statistically when awarding credit.