Comments on: Spinning Straw into Gold: How to do gold standard data right

By: Breck

Breck — Wed, 12 Mar 2008 14:27:23 +0000

Given that we suspect that my and Bob’s recall errors are correlated–meaning we tend to miss the same things–then the question becomes how many annotators do we need to get 100% recall? I made the estmate that I was missing 5% of the recall upon adjudication as was Bob. We need to get a third annotator to decide if the union of me and Bob misses .025% of recall against the third annotation. If so then that supports independence of my and Bob’s recall oversights. If the union annotation misses more than that, then we have evidence that my and Bob’s errors are correlated. All of this has to be wrapped in a meaningful statistical analysis over many iterations but that is the general idea.

It would be a fairly easy experiment to run to see for 20 abstracts when do you start not seeing increased recall with new annotators. I am guessing you would start to see little new found around 4-5 annotaotors but I don’t know.

breck

By: Chris

Chris — Tue, 11 Mar 2008 14:26:15 +0000

Does the need for 90% recall have to do with only using two raters? Imagine using 100 raters, or 1000. How low could the agreement requirements go given larger numbers of annotators? I’m asking because I’ve just blogged about the feasibility of using a web-based app to get raters from all over the world to complete large-scale annotation projects. It may be a pipe dream, even if not a LingPipe dream … rimshot!

By: lingpipe

lingpipe — Mon, 10 Mar 2008 23:12:51 +0000

The real question I have is: is the truth out there? Reporting 90% recall for an annotator assumes the 90% is 90% of something. Philosophically old-fashioned (or naive) practitioners might call this "the truth", but Breck and I studied Quine and have adopted pragmatism as the official semantic theory of Alias-i. What we're really proposing is the adoption of a pragmatic evaluation of recall that bypasses the need for "truth". On a technical note, what I (Bob) would actually like to do is get multiple annotators and try to measure just how correlated the errors are. I'm guessing they'll be highly correlated, and thus we'll need more than two 90% recall annotations to get to 99% recall even measured pragmatically.