LREC 2010 Tutorial: Modeling Data Annotation


Massimo and I are giving our tutorial this afternoon in Malta at LREC 2010. Here’s a link to my slides:

Thanks to Massimo for lots of great suggestions and feedback from when I gave the talk in Rovereto Italy last week (U. Trento), when we talked about it at length between then and now, and on the first few drafts of the slides.

And here’s a link to Massimo’s slides:

Massimo’s part was based on his and Ron’s CL Journal review of kappa-like stats, and the tutorials they’ve given in the past. As such, they’re much more mature. The examples are really tight and to the point.

3 Responses to “LREC 2010 Tutorial: Modeling Data Annotation”

  1. Mark J Says:

    I’m really pleased to see this material getting out there — good work!

    How long until we have a Bayesian replacement for evalb? (smile)


    • lingpipe Says:

      Thanks for the vote of confidence.

      As Andrew Gelman’s always suggesting, it makes sense to do multiple comparisons in a hierarchical model to evaluate bakeoffs.

      At the very least, I’d like to see the bootstrap used for variance estimates rather than making all the erroneous independence assumptions you get with other tests.

      What Massimo and I are worrying about now is how to take his Phrase Detectives coref data and create gold standards. It’s easy to move from binomial to multinomial or ordinal or scalar, but so far we haven’t figured out how to do it with coreference chains. The combinatorics of the sets are rather daunting.

  2. Stuart Moore Says:

    I found the tutorial very useful, thank you.

    I’ve been trying to find suitable paper(s) that explain this to present at our group’s reading club – you give some references in the slides (e.g. Bruce and Wiebe 1999) but you don’t specify exactly which paper, which is making it hard to track them down. Is there any chance you could give a full reference list? Is there a single paper that explains the sensitivity/specificity concept and why it’s a better option than majority voting?

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: