There’s been a back-and-forth on the Computational Linguistics Editorial Board mailing list about how to get people to complete reviews in a timely fashion. I raised the issue that we (reviewers) are all volunteers, so it’s hard to motivate people to do the work. Another ed. board member pointed out that it’s good for your career. Is it?
As Mark Steedman once pointed out to me, the reward for doing a good job on administration is more administrative duties. Hardly the carrot we were looking for to motivate our reviewers.
In my experience, during tenure and promotion at research universities in the U.S., admin per se is universally ignored (or at most given pro forma treatment as part of some onerous template for promotion packages).
On the other hand, general “standing” in the field can be helped by this kind of thing and will come through in reference letters indirectly.
But there’s an asymmetry of information concern here. Only the editor of the journal (Robert Dale at the present time) knows who completes reviews in a timely manner. If I’m an area chair for a conference or on a program committee, I might also see who returns things in a timely manner, and who does a good job.
My feeling is that there’s not much penalty for never agreeing to be on an ed board, and never agreeing to review a paper.