EM’s awfully prone to local maxima in complex problems. Once there’s a reasonable amount of training data, I haven’t seen much in the way of improvement from EM. I’m thinking I’ll try Gibbs sampling instead of EM next; it’s even easier for classifiers than EM, and less prone to get stuck in local optima.

In my experience, scaling up the number of annotators tends to be more useful than running EM. I could be doing something wrong though.

It’s really interesting that so many fields have reinvented aspects of these techniques.

Brendan

They’re also not doing spell checking (or at least doing it with such low recall I haven’t seen it), and they have lots of non-existent pages referenced (as does Google these days).

After two days, I conclude that I can’t use Searchme as a general Google replacement, but it’s still very useful for searching for things like recipes where visual inspection is so much better than snippets. So it’s at least staying on my Firefox pulldown.

Figuring out true paper “quality” given referee scores was one of the applications I’ve been thinking about. I originally thought a simple bootstrap analysis would be interesting, and it would be simple. But now I’ve been thinking more along the lines of linear modeling, perhaps with logistic linking to get the boundedness of the scale.

Thanks for the reference. This is just the kind of thing I was looking for. And looking at the papers that cite it has opened up a vein of this kind of literature.

I’m working on a very similar approach using a Gibbs sampler, which has the nice property of giving me posterior uncertainty estimates of things like confusion matrices.

In searching for your reference, I found this, which is where I was planning to go with the posterior category samples from the fitted models:

Learning with Multiple Labels. Rong Jin and Zoubin Ghahramani.
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.14.8894

I’ll post the details next week when I’m done writing up the paper.

We first assume known annotator accuracy (initialized, say, to perfect accuracy), and based on that we compute the most likely labels.

Then, based on the estimated labels, we can estimate the labelers accuracy.

By iterating, we can converge to some good estimates of the labels, and generate a confusion matrix for each annotator.

Take a look at
Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm
A. P. Dawid and A. M. Skene
Applied Statistics, Vol. 28, No. 1 (1979), pp. 20-28
http://www.jstor.org/pss/2346806

For their comparison with SbA (syllabification by analogy), they used about 15K training instances:

#train=14696 #test=24921
Whole word accuracy = 0.8789 (Struct SVM = 0.8999)
Per hyph precision = 0.9316
Per hyph recall = 0.9305
Per decision error = 0.0316 (0.0242)

Their structured SVM approach is significantly better than the simple character LM noisy channel approach.

With their 60K train, 5K test, and our default English params (8-grams with default 8.0 interpolation), we get these results

#train=60000 #test=5000
accuracy=0.9538 (struct SVM = 0.9565)
Per hyphenation prec = 0.9732
per hyphenation recall = 0.9739
per tagging decision error = 0.0124
accuracy=0.9544

With our best cross-validating result on the training data, running on their train/test split, we get this:

whole word accuracy=0.9544 (struct SVM = 0.9565)
per hyphenation precision=0.9745
per hyphenation recall=0.9742

Here the results are much closer. We’ve seen this pattern before.

The errors have an interesting pattern where stress is a determining factor. Lots of words have different hyphenation with different forms, such as CHER-ub vs. CHER-u-bim vs. che-RU-bic. It sure seems that building a joint model of pronunciation (including lexical stress) and syllabification/hyphenation would be a big win. There’s also ambiguity in syllable reduction, such as the word "mayor" being one syllable or two, and words like "aqualung" having variant pronuncations ak-wuh-luhng vs. ah-kwuh-lung (note which syllable the “q” sound shows up in).