Does Reviewing (or Other Service) Help Your Career?

by

There’s been a back-and-forth on the Computational Linguistics Editorial Board mailing list about how to get people to complete reviews in a timely fashion. I raised the issue that we (reviewers) are all volunteers, so it’s hard to motivate people to do the work. Another ed. board member pointed out that it’s good for your career. Is it?

As Mark Steedman once pointed out to me, the reward for doing a good job on administration is more administrative duties. Hardly the carrot we were looking for to motivate our reviewers.

In my experience, during tenure and promotion at research universities in the U.S., admin per se is universally ignored (or at most given pro forma treatment as part of some onerous template for promotion packages).

On the other hand, general “standing” in the field can be helped by this kind of thing and will come through in reference letters indirectly.

But there’s an asymmetry of information concern here. Only the editor of the journal (Robert Dale at the present time) knows who completes reviews in a timely manner. If I’m an area chair for a conference or on a program committee, I might also see who returns things in a timely manner, and who does a good job.

My feeling is that there’s not much penalty for never agreeing to be on an ed board, and never agreeing to review a paper.

4 Responses to “Does Reviewing (or Other Service) Help Your Career?”

  1. Ken Says:

    In my area, medical statistics, the leading methodological journal Statistics in Medicine has a large number of associate editors, so you do get to deal with a leading statistician in your particular area.

    Statistical reviewing for medical journals is completely different, you don’t know anyone, but one of the reasons for receiving it may be that it references some of your methodology. In which case it is nice to see whether they do it properly. Otherwise it is a boring way of spending time and definitely not much point.

  2. Rich W Says:

    To me, it’s valuable way (though definitely not the most important way) to stay current and relevant in the field. The process forces me to be able to read, understand and evaluate new work. This skill is helpful since I’m the only one who works in NLP where I work, so there are no reading groups or the like to help keep me current. Also, I agree with Mark that doing (largely) thankless reviewing does lead to more administrative work — but often that work is more important or visible (ie program or conference chair, workshop organizer, etc). For younger faculty, I think those more visible administrative roles would not be ignored at promotion time (but my view of what goes on at research universities is obviously flawed since I’m not at one). I expect that even for more senior faculty, important visible administrative roles are helpful because people do keep coming forward to do them!

  3. Anonymous Says:

    I definitely had the feeling that I was wasting my (and everybody else’s) time when I was on an editorial board. I did not have any more say in what sorts of papers would get accepted, only more reviewing duties, including what I’ll call “review churn”. By which I mean the seemingly endless cycle of submission, ho-hum reviews, a decision to resubmit with minor to substantial revisions, followed by one or more similar cycles. I got quickly fed up by this and other dysfunctional aspects of the reviewing process.

    My current feeling is that we need to move to purely electronic open-access journals, if only for the sake of the readers and reviewers. Then the cost of publishing a 10-page paper is virtually the same as publishing a 50-page paper; and the cost of publishing a great paper is more or less the same as the cost of publishing a so-so paper — or rather, we’re passing the burden of sifting and sorting on to the reader. As a consequence, we would be able to set the bar for acceptance pretty low: anything that follows basic standards and practices and/or is not embarrassing can and should be published. The real work begins after publication, when readers will rate and review papers, have Wiki-style discussions, rejoinders, etc. Hopefully, over time, the good papers would percolate to the top. For this to happen, it’s best to have input from as many people as possible, as opposed to three or four reviewers. In such a world, the role of the reviewers or ed board members is more like the role of admins on community boards or on Wikipedia, namely to keep the vandals at bay and the community functional. Reviewers and editors wouldn’t have to be guardians of standards and quality, which would greatly simplify the entire process.

  4. lingpipe Says:

    @Anonymous Wow, that eerily echoes my own feelings. I’m particularly peeved about the revise/resubmit cycle, and have been arguing against giving any papers that decision. It turns out that many papers are also lost this way, because authors don’t want to undertake revisions that are often on the scale of the original paper, especially given uncertainty of final publication.

    I do feel like I get some say in which papers get published, but the final decision’s not mine. I was so fed up with COLING reviewing that I stopped doing it years ago — the papers accepted never bore any relation to my reviews (and back in the day, reviewers would get 10 papers/conference). I was also fed up with the one NSF panel I was on because the program manager basically overrode the committee’s ratings; I felt like a tool being used to put an imprimatur of peer review on an fundamentally biased and utterly subjective process.

    Which brings me to your last point. I completely agree that we should stop trying to cut such a fine line between acceptable and non-acceptable papers. At conferences and in journals. First, I think the sensitivity and specificity of the reviewing process is much lower than everyone in favor of low acceptance rates thinks it is. Second, I think it’s also biased toward conservative, easy to numerically evaluate papers. I tend to enjoy the more out-there workshop papers, and what I really like is hour-long talks by people on what they want to work on next.

    Finally, I think word-of-mouth, aka network centrality, will sort the wheat from the chaff. That’s how I find papers anyway — I don’t just assume everything published in Computational Linguistics is good because of our editorial process.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s