Comments on: Dolores Labs’ Text Entailment Data from Amazon Mechanical Turk

By: Cade

Cade — Fri, 15 Apr 2011 02:26:08 +0000

Cujcgw YMMD with that awnesr! TX

By: lingpipe

lingpipe — Fri, 19 Sep 2008 16:30:13 +0000

Yes, that’s exactly what happens. I just ran the data removing any annotator with an estimated sensitivity or specificity less than 50%. It makes almost no difference for the model-based approach, but makes a huge difference for majority voting, which winds up half way between the model-based results and unfiltered voting results if you remove the outliers. I was surprised the results were this robust. I’ll have to add another blog entry. I’m on the road now, so I might not get this stuff posted for a few days.

There’s a minor identifiability problem with the model with bad annotators in that you can get the same results with 20% accurate annotators being wrong a lot and 80% accurate annotators being right a lot. So when you open up the possibility of less than 50% sensitivity or specificity, you have to run until you don’t get these degenerate solutions (that is, until the chains mix the way the Bayesians like to see them during the Gibbs samples).

By: Brendan O'Connor

Brendan O'Connor — Fri, 19 Sep 2008 08:52:39 +0000

This is great. I'm struck by how small the worker model residuals are. I have the exact numbers on my end to round out your comparisons of the techniques. I put it back on the other post. On removing junk annotations: the model is figuring out that junk annotators are actually junk, so they should already be exerting very weak influence on the posterior of the labels, and therefore a weak influence the sens/spec estimates of other workers. So if I had to bet, I'd say taking out junk annotators won't change the model's inferences by very much? Though junk annotators do matter a lot; in my other comment on the other post i found that throwing out the junk makes naive voting perform as well as anything we've seen so far... Hooray for open data/software/etc. This all sounds like a good kind of science to me.

By: Opening up Academic Research on IR and Machine Learning

Opening up Academic Research on IR and Machine Learning — Thu, 18 Sep 2008 17:09:13 +0000

[...] Pedersen for finally saying out loud (in the latest issue of Computational Linguistics, thanks to Bob Carpenter for the pointer) what I’ve long thought about academic publications on topics like [...]