Whew. I spent most of the weekend finishing this off. It’s one stop shopping for all the info on models and applications I’ve been blogging about (hopefully with enough introductory material that it’ll be comprehensible even if you don’t know anything about latent data models, multilevel models, or Bayesian inference):
- Carpenter, Bob. 2008 Multilevel Bayesian Models of Categorical Data Annotation. Technical Report. Alias-i.
It’s 52 pages, but it’s mostly figures. As usual, any feedback here or to my e-mail (
email@example.com) would be greatly appreciated. Here’s the abstract:
This paper demonstrates the utility of multilevel Bayesian models of data annotation for classiers (also known as coding or rating). The observable data is the set of categorizations of items by annotators (also known as raters or coders) from which data may be missing at random or may be replicated (that is, it handles fixed panel and varying panel designs). Estimated model parameters include the prevalence of category 1 outcomes, the “true” category of each item (the latent class), the accuracy in terms of sensitivity and specicity of each annotator (latent annotator traits), the difficulty of each item (latent item traits). The multilevel parameters represent the average behavior and variances among the annotators and the items. We perform inference with Gibbs sampling, which approximates the full posterior distribution of parameters as a set of samples. Samples from the posterior category distribution may be used for probabilistic supervision and evaluation of classiers, as well as in gold-standard adjudication and active learning. We evaluate our approach with simulated data and two real data sets, including data for which a prior “gold standard” exists.
All the code (R, BUGS, with some Java for munging), all the data, and the source for the paper are available from the LingPipe sandbox in the project
hierAnno; see the sandbox page for information on checking it out, or just use this command (you’ll need to have CVS installed):
cvs -d :pserver:firstname.lastname@example.org:/usr/local/sandbox checkout hierAnno
Next up, I’m hoping to collect some named entity data in time to write this all up for a NAACL/HLT 2009 submission, so I’m especially keen to get feedback before then.