I was scooped by roughly 30 years in the epidemiology literature for a simple model, and by 10 years in epidemiology literature for both Bayesian models. They don’t seem to have ever estimated their hierarchical priors (the beta params), or dealt with partial data, but those are pretty simple additions.

That’d be about 10 years in the CL literature — Bruce and Wiebe applied the Dawid and Skene 1979 scheme in 1999, so I’d say I was scooped on the notion of trying to induce a gold standard by about 10 years.

The really cool stuff coming out now is the EMNLP paper from Dolores Labs and friends on using the Amazon Mechanical Turk to do annotations. I’m about to dump out results of the binomial tagging models on that data.

Is this post an indirect way of saying that you were scooped on the annotation evaluation model you posted on the 5th Sept? If so, is there a specific paper on this kind of annotation evaluation that you can recommend? BTW, I think your model is very nice, as it gives us a way of estimating the accuracy of the annotations, which is what we really want to know.

Mark