In case you didn’t see the link on Slashdot, Google was supplying maps of the American state Georgia when they should’ve been linking to the Caucasian country Georgia. As Homer (the Simpson, not the classical Greek poet, the Alaskan city, the Illinois city, the American painter, or the tunnel in New Zealand) would say, “D’oh!.”

Slashdot was just picking up the story from Valleywag. Google didn’t take this lying down; the current page shows a map centered on Vienna, which if you click on it, takes you to New York, saying “U.N.”.

The problem is that linking textual mentions to database entries is non-trivial, even for the relatively simple problem of geo-location. This is the business Metacarta is in, and users we’ve talked to say they do a very good job of it.

You see similar problems for products, as in the app formerly known as Froogle or ShopWiki, as the Slashdot story pointed out in the case of yet another search engine mismatching product photos.

We ran into this problem trying to find rap artists in text, who have names like “The Game.” And we’re battling the problem in our ongoing NIH project on linking genes and protein mentions in text to Entrez-Gene.

I’ve noted before that the NY Times site runs into problems in cases such as distinguishing the Pittsburgh suburb named Mount Lebanon from the area of middle Eastern country Lebanon. In general, text matching doesn’t work real well by itself. You run into false positives by linking the Pittsburgh suburb to the middle East, but you get false negatives if you just don’t link “Mount Lebanon”. But the Times has full editorial control, so they can catch these problems and fix them manually during copy editing.

To solve this problem automatically with better results than plain text matching, we need context, which is available on the web in the form of both texts and links. We discuss this in our clustering tutorial, which uses context to disambiguate multiple John Smiths in the news, and in our white paper on gene linkage. This basically reduces the linkage problem to that of word sense disambiguation. The only real problem (besides the remaining difficult to disambiguate cases) is that it’s relatively storage and compute intensive to use context compared to a simple dictionary matcher.

Antonio Valderrabanos of Bitext.com gave a talk at NYU today and he compared the current search/NLP strategies by information providers to  humanity’s hunter/gatherer stage and offered a vision of information farming. I kept having images of Google’s web spider out digging roots and chasing animals with a pointed stick wearing a grubby little loin cloth. Then I would switch to images of a farm stocked supermarket with well organized shelves, helpful clerks and lots of choice.

The analogy brought up a strong bias that I have in applying natural language processing (NLP) to real word problems– I generally assume that the software must encounter text as it occurs in the “wild”–after all it is what humans do so well and we are in the business of emulating human language processing right?

Nope, not on the farm we’re not. On the farm we use NLP help to enhance information that was never a part of standard written form. We use NLP to suggest and assign meta tags, connect entities to databases of concepts and create new database entries for new entities. These are things that humans are horrible at but humans are excellent at choosing from NLP driven suggestions– NLP is pretty good at suggestions. So NLP is helping create the tools to index and cross reference at the concept level all the information in the supermarket. Humans function as filters of what is correct. At least initially.

As the information supermarket gets bigger, the quality of the NLP (machine learning based) will get better, perhaps good enough to start automatically bringing in “wild” information with decent concept indexing and meta tagging. A keyword index is crude yet effective tool but an inventory system it is not and that is what we need to advance to the next level of information access.

I (Bob) have been looking at the inter-annotator agreement and gold-standard adjudication problems. I’ve also been hanging out with Andrew Gelman and Jennifer Hill thinking about multiple imputation, which has me studying their regression book, which covers the item-response (Rasch) model, and Gelman et al.’s book, which describes the the beta-binomial model. After thinking about how these tools could be put together, I believed I might have a novel approach to modeling inter-annotator agreement. I got so excited I went and wrote a 30 page paper based on a series of simulations in R and BUGS, which may still turn out to be useful as an introduction to these approaches. It’s already been useful in giving me practice in building hierarchical models, simulations, and generating graphics from BUGS and R. I even analyzed one of our customer’s real annotated data sets (more on that to follow, I hope, as they had 10 annotators look at 1000 examples each over half a dozen categories).

Alas, the epidemiologists have beaten me to the punch. They generalized the item-response model in exactly the way I wanted to and have even implemented it in WinBUGS. In fact, they even used pretty much the same variable names! It just goes to show how much people with the same tools (hieararchical Bayesian modeling, logistic regression, beta-binomial distributions, 0/1 classification problems, sensitivity vs. specificity distinctions) tend to build the same things.

If you can only read one paper, make it this one (if you can find it; I had to schlep up to Columbia where I can download papers on campus):

• Albert, Paul S. and Lori E. Dodd. 2004. A Cautionary Note on the Robustness of Latent Class Models for Estimating Diagnostic Error without a Gold Standard. Biometrics 60:427-435.

It cites most of the relevant literature other than (Dawid and Skene 1979), who, as far as I can tell, first introduced latent class models into this domain using EM for estimation.

To make a long story short, the model is quite simple in modern sampling/graphical notation, and just about as easy to code up in BUGS. Here’s the model without accounting for missing data. First the variable key:

     I       # of items to classify
     J       # of annotators
     pi      overall prevalence of category 1
     c[i]    true category of item i
     d[i]    difficulty of item i
     a[0,j]  specificity of annotator j
     a[1,j]  sensitivity of annotator j
     x[i,j]  annotation of item i by annotator j

Recall that sensitivity is accuracy on positive cases, which is just recall, TP/(TP+FN), whereas specificity is just accuracy on negative cases, TN/(TN+FP). Precision is TP/(TP+FP), but that doesn’t account for TN cases, and is thus incomplete as a full probability specification when paired with recall. That’s why ROC curves, which plot specificity vs. sensitivity, are more popular than precision-recall curves in the rest of the civilized world.

Other than the annotations x[i,j], all other variables are unknown. That includes the prevalence pi, the true categories c, the annotator specificities and selectivities a[0] and a[1].

The sampling model without the noninformative priors:

  c[i]  ~ Bernoulli(pi)
  d[i]  ~ Normal(0,1)
a[0,j]  ~ Beta(alpha[0],beta[0])
a[1,j]  ~ Beta(alpha[1],beta[1])
x[i,j]  ~ Bernoulli(inv-logit(logit(a[1,j]) - d[i]))     if c[i] = 1
          Bernoulli(1 - inv-logit(logit(a[0,j]) - d[i])) if c[i] = 0

That’s it. Not that complex as these things go. Scaling the difficulties to have 0 mean and variance 1 identifies the scale of the model; as Gelman and Hill describe in their book, there are lots of ways this can be done, including only scaling the mean of the difficulties to be 0.

There are lots of different priors that could be put on what’s here the logit(a[m,j]) terms. The more traditional thing to do in this kind of model is to use a normal prior. In any case, you’re not going to be able to estimate the priors for specificity and selectivity with only a handful of annotators.

There’s a simplified version of this model mentioned in Dodd and Albert where items are divided into easy and regular cases, with the easy cases having all annotators agree and regular cases having annotators respond independently according to their own specificity and selectivity.

The point of the Albert and Dodd paper cited above wasn’t to introduce these models, but to evaluate a range of them one against the other by simulating data in one model and evaluating it in the other. They also evaluated real data and saw how it fit differently in the different models.

I should also point out that the following paper mentions the Dawid and Skene model in the computational linguistics literature:

• Bruce, Rebecca F. and Janyce M. Wiebe. 1999. Recognizing subjectivity: a case study of manual tagging. Natural Language Engineering 1:1-16.

Bruce and Wiebe even talk about using the posterior distribution over true categories as a proxy for a gold standard, which seems to be the right way to go with this work. But they use a single latent variable (roughly problem difficulty) and don’t separately model annotator specificity from selectivity, which is critical in both the simulations and real world data analyses I’ve done recently.

The depressing conclusion for NLP and other applications of classifiers is that it’s clear that with only 3 annotators, it’s going to be impossible to get a gold standard of very high purity. Even with 5 annotators, there are going to be lots of difficult cases.

The other applications besides inter-annotator agreement that I’ve run across in the past couple of days are educational testing, epidemiology of infections and evaluating multiple tests (e.g. stool inspection and serology), evaluations of health care facilities in multiple categories, evaluations of dentists and their agreement on caries (pre-cavities), adjustments to genome-wide prevalence assertions, and many more.

I’m repeating pretty much verbatim a comment I left on Páraic Sheridan’s blog, Returned Emigrant in a response to his post Search is not a solved problem.

Discussing whether search is a solved problem reminds me of a talk Hynek Hermansky gave at ICSLP in Sydney in ‘98 on why speech recognition isn’t a solved problem. Hynek made an analogy to flight, where at any point between balloons and jet airplanes, flight might’ve been considered solved.

Searchme.com is great. They provide reduced resolution page views with overlayed snippets, and it’s fast. This just feels right for navigation searches (e.g. typing “lingpipe” to try to find our home page or blog). And the page views add more value than I could’ve imagined to the snippets. Like Páraic, I’m so enamored of it that it’s replacing Google as my default search engine on Firefox (just click on the link in the upper right of searchme.com’s home page). I sure hope they can scale as more people find out about them.

Cuil.com, by focusing on recall (and marketing), seems less useful, even if they get the bugs ironed out. Despite the fact that we’re focusing on recall for genomics information extraction tasks, I’ve never felt recall was an issue for most web searches. I could use more approximate and contextual matching, perhaps, but the index size has never seemed an issue.

I miss Excite.com, which used to run TF/IDF rather than social-network-based search ranking algorithms. I missed Excite even as I was starting to use Google for many searches. But then again, if Excite had been successful, we wouldn’t have the Apache Lucene search engine.

PowerSet.com was focusing on some kind of precision and question answering (and marketing), which I also felt was of questionable value (for me as a searcher; it clearly worked for their VCs and founders) compared to using Google. Plus, they never showed (at least to the public) that their tech scaled either in complexity (different page types, multiple pages for entities) and size (number of web pages).

My last blog post, Good Kappa’s Not Enough, summarized Reidsma and Carletta’s arguments that a good kappa score is not sufficient for agreement. In this post, I’d like to point out why it’s not necessary, either. My real goal’s to refocus the problem on discovering when a gold standard can be trusted.

Suppose we have five annotators who are each 80% accurate for a binary classification problem whose true category distribution is 50-50. Now let’s say they annotate an example item (1,1,1,1,0), meaning annotator 1 assigns category 1, annotator 2 assigns category 1, up through annotator 5 who assigns category 0. What can we conclude about the example? Assuming the errors are independent (as kappa does), what’s the likelihood that the example really is of category 1 versus category 0? Bayes’ rule lets us calculate:

     p(1|(1,1,1,1,0)) proportional to p(1) * p((1,1,1,1,0)|1)
                            = 0.5 * 0.8^4 * 0.2^1

     p(0|(1,1,1,1,0)) proportional to 0.5 * 0.8^1 * 0.2^4

     p(1|(1,1,1,1,0)) = (0.8^4 * 0.2^1)
                            / (0.8^4 * 0.2^1 + 0.8^1 * 0.2^4)
                       = 98.5%

Recall the definition of kappa:

     kappa = (agree - chanceAgree) / (1 - chanceAgree)

If errors are distributed randomly, agreement will be around 0.8^2 + 0.2^2 = 0.68 in a large sample, and the chance agreement will be 0.5^2 + 0.5^2 = 0.5, for a kappa value of (0.68-0.5)/(1-0.5)=0.36. That’s a level of kappa that leads those who follow kappa to say “go back and rethink your task, your agreement’s not high enough”.

Unfortunately, with 80% per-annotator accuracy, we only expect 74% or so of the examples to have a 4:1 or 5:0 vote by 80% accurate annotators (74% = 5 * 0.8^4 0.2^1 + 0.8^5).

I believe the question we really care about is when we can trust an annotation enough to put it in the gold standard. So let’s say we have two 80% accurate annotators and the true category is 1. The likelihood of various annotation outcomes are:

     p(1,1) = 0.64     p(0,1) = 0.16
     p(0,0) = 0.04     p(1,0) = 0.16

So clearly two annotators aren’t enough to be confident to the 99% level. We’d need 90% accurate annotators for that. But what about three annotators? The chance of three 80% annotators agreeing by chance is only 0.8%. And in 51.2% of the cases, they will agree and be right. So we use a minimum of three annotators, and if they agree, go on.

If they disagree, we need to bring in more annotators until we’re confident of the outcome. When we get to a four out of five vote, as in our first example, we’re confident again. But even 3/4 agreement is still pretty weak, yielding only a 94% chance that the agreed upon value is correct.

Of course, this line of reasoning supposes we know the annotator’s accuracy. In practice, we can’t evaluate an annotator’s accuracy because we don’t know the true labels for items.

I stumbled across Reidsma and Carletta’s Reliability measurement without limits, which is pending publication as a Computational Lingusitics journal squib (no, not a non-magical squib, but a linguistics squib).

The issue they bring up is that if we’re annotating data, a high value for the kappa statistic isn’t enough to guarantee what they call "reliability". The problem is that the disagreements may not be random. They focus on simulating the case where an annotator over-uses a label, which results in kappa way overestimating reliability when compared to performance versus the truth. The reason is that the statistical model will be able to pick up on the pattern of mistakes and reproduce them, making the task look more reliable than it is.

This discussion is similar to the case we’ve been worrying about here in trying to figure out how we can annotate a named-entity corpus with high recall. If there are hard cases that annotators miss (over-using the no-entity label), random agreement assuming equally hard problems will over-estimate the "true" recall.

Reidsma and Carletta’s simulation shows that there’s a strong effect from the relationship between true labels and features of the instances (as measured by Cramer’s phi).

Review of Cohen’s Kappa

Recall that kappa is a "chance-adjusted measure of agreement", which has been widely used in computational linguistics since Carletta’s 1996 squib on kappa, defined by:

kappa = (agreement - chanceAgreement)
           / (1 - chanceAgreement)

where agreement is just the empirical percentage of cases on which annotators agree, and chanceAgreement is the percentage of cases on which they’d agree by chance. For Cohen’s kappa, chance agreement is measured by assuming annotators pick categories at random according to their own empirical category distribution (but there are lots of variants, as pointed out in this Artstein and Poesio tech report, a version of which is also in press at The Journal of Kappa Studies, aka Computational Linguistics). Kappa values will range between -1 and 1, with 1 only occurring if they have perfect agreement.

I (Bob) don’t like kappa, because it’s not estimating a probability (despite being an arithmetic combination of [maximum likelihood] probability estimates). The only reason to adjust for chance is that it allows one, in theory, to compare different tasks. The way this plays out in practice is that an arbitrary cross-task threshold is defined above which a labeling task is considered "reliable".

A final suggestion for those using kappa: confidence intervals from bootstrap resampling would be useful to see how reliable the estimate of kappa itself is.

As part of our NIH grant, we’re working on the database linkage problem from gene/protein mentions in MEDLINE to database entries in EntrezGene. Basically, it’s what the biologists call "gene normalization", and was the basis of Biocreative Task 2.

I can summarize the problem we’re having with a simple example. We’d like to classify all 17M or so entries in MEDLINE as to whether they’re about genomics or not. EntrezGene provides links to 200K citations that are about particular genes, so we have a pile of articles about genomics (making up about 300 million characters). What we don’t have is any negative training data.

So my question is: how do I build a classifier for articles about genomics versus those that are not about genomics?

The job running in the background giving me time to write this post is generating cross-validation on cross-entropy rates for all of these 200K citations. That is, I train a character-level language model on 180K citations and use it to evaluate the other 20K, for all possible choices. This gives me a received set of expected scores for positive examples (assuming there’s no bias in that 200K set, which there is in terms of recency and particular gene focus, not to mention focus on human genomics). I’m going to plot this and see what the curve looks like. Empirically, we can then set a threshold that would accept 99% of the articles we have. Unfortunately, I have no idea how well this’ll work in practice at rejecting the articles that aren’t about genomics.

For the genomics/non-genomics problem, we can just annotate a few thousand examples; it’ll only take a day or two.

The real problem is that we want to build models to classify contexts for the 30K or so human gene entries in Entrez. Some of them have a handful of example docs, some have hundreds. We’re going to pull out articles with potential mentions, then filter with the classifier. It’s related to what we did in Phase I of our grant and reported in our gene linkage white paper. In that setting, we can generate candidate docs using approximate matching of aliases, then use the scores to rank the possible docs according to their language model scores against the known articles for the gene in question. This is great in a search context, but doesn’t give us a go/no-go decision point, which we need for some of our downstream applications.

If anyone knows how to tackle this problem, I’d love to hear about it. I might even implement it as part of LingPipe if the idea’s simple and general enough.

Noisy channel models with very simple deterministic channels can be surprisingly effective at simple linguistic tasks like word splitting. We’ve used them for Chinese word segmentation (aka word boundary detection, aka tokenization), not to mention spelling correction.

In this blog entry, I’ll provide a first look at our forthcoming tutorial on hyphenation. The hyphenation problem is that of determining if a hyphen can be inserted between two positions in a word. Hyphenation is an orthographic process, which means it operates over spellings. In contrast, syllabification is a phonological (or phonetic) process, which means it operates over sounds. Hyphenation roughly follows syllabification, but is also prone to follow morphological split points.

The hyphenation problem isn’t even well-defined on a per-word level. There are pairs such as num-ber (something you count with) and numb-er (when you get more numb) that have the same spelling, but different pronunciations and corresponding hyphenations depending on how they are used. I’ll just ignore this problem here; in our evaluation, ambiguities always produce at least one error.

The Noisy Channel Model

The noisy channel model consists of a source that generates messages and a noisy channel along which they are passed. The receiver’s job is to recover (i.e. decode) the underlying message. For hyphenation, the source is a model of what English hyphenation looks like. The channel model deterministically removes spaces. The receiver thus needs to figure out where the hyphens should be reinserted to recover the original message at the source.

The training corpus for the source model consists of words with hyphenations represented by hyphens. The model is just a character language model (I used 8-grams, but it’s not very length sensitive). This gives us estimates of probabilities like p("che-ru-bic") and p("cher-u-bic"). Our channel model just deterministically removes spaces, so that p("cherubic"|"che-ru-bic") = 1.0.

To use the noisy channel model to find a hyphenation given an unhyphenated word, we just find the hyphenation h that is most likely given the word w, using Bayes’s rule in a maximization etting: ARGMAXh p(h|w) = ARGMAXh p(w|h)*p(h). Because the channel probabilities p(w|h) are always 1.0 if the characters in w match those in h, this reduces to finding the hyphenation h yielding character sequence w which maximizes p(h). For example, the model will estimate "che-ru-bic" to be a more likely hyphenation than "cher-u-bic" if p("che-ru-bic") > p("cher-u-bic").

Decoding is fairly efficient for this task, despite using a high-order n-gram language model, because the channel bounds the combinatorics by only allowing a single hyphen to be inserted at each point; dynamic programming into language model states can reduce them even further.

English Evaluation

So how do we test how well the model works? We just became members of the Linguistic Data Consortium, who distribute Baayen, Piepenbrock and Gulikers’ CELEX 2 corpus. CELEX is a great corpus. It contains pronunciations, syllabifications, and hyphenations for a modest sized dictionary of Dutch, English and German. For instance, there are 66,332 distinct English words in the corpus (there are many more entries, but these constitute duplicates and compounds whose constituents have their normal hyphenations). These 66,332 word have 66,418 unique hyphenations, meaning 86 of the words lead to ambiguities. This is about 1/10th of a percent, so I won’t worry about it here.

I evaluated with randomly partitioned 10-fold cross-validation. With the noisy channel model above, LingPipe had a 95.4% whole word accuracy, with a standard deviation of 0.2%. That means we got 95.4% of the words completely correctly hyphenated. We can also look at the 111,521 hyphens in the corpus, over which we had a 97.3% precision and a 97.4% recall. That is, we missed 2.6% true hyphenation points (false negatives), and 2.7% of the hyphenations we returned were spurious (false positives). Finally, we can look at per-decision accuracy, for which there were 482,045 positions between characters, over which we were 98.8% accurate.

Forward and Backward Models

But that’s not all. Like HMMs, there’s a degree of label bias in a left-to-right language model. So I reversed all the data and built a right-to-left (or back-to-front) model. Using n-best extraction, I ran this two ways. First, I just added the score to the forward model to get an interpolated score. Somewhat surprisingly, it behaved almost identically to the forward-only model, with slightly lower per-hyphen and per-decision scores. More interestingly, I then ran them in intersect mode, which means only returning a hyphen if it was in the first-best analysis of both the left-to-right and right-to-left models. This lowered per-word accuracy to 94.6% (from 95.4%), but raised precision to 98.0% (from 97.3%) while only lowering recall to 96.7% (from 97.4%). Overall, hyphenation is considered to be a precision business in application, as it’s usually used to split words across lines in documents, and many split points might work.

Is it State of the Art?

The best results I’ve seen for this task were reported in Bartlett, Kondrak and Cherry’s 2008 ACL paper Automatic Syllabification with Structured SVMs for Letter-To-Phoneme Conversion, which also received an outstanding paper award. They treated this problem as a tagging problem and applied structured support vector machines (SVMs). On a fixed 5K testing set, they report a 95.65% word accuracy, which is slightly higher than our 95.4%. The 95% binomial confidence intervals for 5000 test cases are +/- 0.58% and our measured standard deviation was .2%, with results ranging from 95.1 to 95.7% on different folds. Their paper also tackled pronuncation, for which their hyphenation was only one feature.

German and Dutch are easier to hyphenate than English. Syllabification in all of these languages is also easier than hyphenation.

But is it better than 1990?

Before someone in the audience gets mad at me, I want to point out that we could follow Coker, Church and Liberman (1990)’s lead in reporting results:

The Bell Laboratories Text-to-Speech system, TTS, takes a radical dictionary-based approach; dictionary methods (with morphological and analogical extensions) are used for the vast majority of words. Only a fraction of a percent (0.5% of words overall; 0.1% of lowercase words) are left for letter-to-sound rules.

Although this insight won’t get us into NIPS, it’s how we’d field an application.

While we’re on the subject of tokenization, every time I (Bob) use Lucene, I’m struck by its lack of a tokenizer factory abstraction.

Lucene’s abstract class TokenStream defines an iterator-style method next() that returns the next token or null if there are no more tokens.

Lucene uses the abstract class Token for tokens. A Lucene token contains a string representing its text, a start and end position, and a lexical type which I’ve never seen used. Because LingPipe has to handle many tokens quickly without taxing garbage collection, it doesn’t create objects for them beyond their string texts. But that’s the subject of another blog entry.

A Lucene Document is essentially a mapping from field names, represented as strings, to values, also represented as strings. Each field in a document may be treated differently with respect to tokenization. For instance, some might be dates, others ordinary text, and others keyword identifiers).

To handle this fielded document structure, the Lucene class analysis.Analyzer maps field names, represented as strings, and values, represented as instances of java.io.Reader, to token streams. The choice of Reader for values is itself puzzling because it introduces I/O exceptions and the question of who’s responsible for closing the reader.

Lucene overloads the analyzer class itself to provide the functionality of LingPipe’s tokenizer factory. Lucene classes such as SimpleAnalyzer and CJKAnalyzer return the same token stream no matter which field is specified. In other words, the field name is ignored.

What would be useful would be a Lucene interface analysis.TokenStreamFactory with a simple method TokenStream tokenizer(CharSequence input) (note how we’ve replaced the analyzer’s reader input with a generic string). Then analyzers could be built by associating token stream factories with fields. This would be the natural place to implement Lucene’s simple analyzer, CJK analyzer, and so on. The current analyzer behavior is then easily derived with an analyzer which sets a default token stream factory for fields.

We’re now running into a problem we’ve run into before: so-called “intelligent” tokenization. The earliest version of this of which I’m aware is the Penn Treebank tokenization, which assumes sentence splitting has already been done. That way, the end-of-sentence punctuation can be treated differently than other punctuation. Specifically, “Mr. Smith ran. Then he jumped.” gets split into two sentences, “Mr. Smith ran.” and “Then he jumped.”. Now the fun starts. The periods at the end of a sentence are split off into their own token. The period after “Mr” remains, so the tokens are “Mr.”, “Smith”, “ran” and “.”. Note that the Treebank tokenizer also replaces double quotes with either left or right LaTex-style quotes, so there’s no way to reconstruct the underlying text from the tokens. Like many other projects, they also throw away the whitespace information in the data, so there’s no way to train something to do tokenization that’s whitespace sensitive because we just don’t have the whitespace. That’s what you get for releasing data like “John/PN ran/V ./PUNCT” — you just don’t know if there was space between that final verb “ran” and the full stop “.”. You’ll also see their script builds in all sorts of knowledge about English, such as a handful of contractions, so that “cannot” is split out into two tokens, “can” and “not”.

The most recent form of “intelligent” tokenization I’ve seen is perpetrated by UPenn, this time as part of their BioIE project. There, the data’s not even consistently tokenized, because they left it to annotators to decide on token boundaries. They then use statistical chunkers to uncover the tokenizations probabilistically.

Google’s n-gram data is also distributed without a tokenizer. It looks very simple, but there are lots of heuristic boundary conditions that make it very hard to run on new text. Practically speaking, the data’s out of bounds anyway because of its research-only license. Unlike the Penn Treebank or French Treebank, there’s no option to buy commercial licenses.

I’ve just been struggling with the French Treebank, which follows the Penn Treebank’s lead in using “intelligent” tokenization. The problem for us is that we don’t know French, so we can’t quite read the technical docs (in French), nor induce from the corpus how tokenization was done.

This is all terribly problematic for the “traditional” parsing model of first running tokenization, then running analysis. The problem is that the tokenization depends on the analysis and vice-versa. At least with the Penn approach, there’s code to do their ad hoc sentence splitting and then their ad hoc heuristic tokenization. It may not be coherent from an English point of view (a handful of contractions will be split; others won’t), but at least it’s reproducible.

Our own approach (in practice — in theory we can plug and play any tokenizer that can be coded) has been to take very fine-grained tokenizations so that the tokenization would be compatible with any old kind of tagger. Our HMM chunker pays attention to tokenization, but the rescoring chunker uses whitespace and longer-distance token information.

At the BioNLP workshop at ACL 2008, Peter Corbett presented a paper (with Anne Copestake) on Cascaded Classifiers for Confidence-Based Chemical Named Entity Recognition. It’s a really neat paper that addresses issues of confidence estimation, and particularly trading precision for recall (or vice-versa). But they weren’t able to reproduce our 99.99% recall gene/protein name extraction. During the question period, we got to the bottom of what was going on, which turned out to be intelligent tokenization making mistakes so that entities weren’t extractable because they were only parts of tokens. I’m hoping Peter does the analysis to see how many entities are permanently lost due to tokenization errors.

So why do people do “intelligent” tokenization? The hope is that by making the token decisions more intelligently, downstream processing like part-of-speech tagging is easier. For instance, it’s difficult to even make sense of assigning part-of-speech tags to three tokens derived from “p-53″, namely “p”, “-” and “53″. Especially if you throw away whitespace information.

Tokenization is particularly vexing in the bio-medical text domain, where there are tons of words (or at least phrasal lexical entries) that contain parentheses, hyphens, and so on. This turned out to be a problem for WordNet).

In some sense, tokenization is even more vexing in Chinese, which isn’t written with spaces. To get around that problem, our named-entity detector just treats each character as a token; that worked pretty well for our entry in the SIGHAN 3 bakeoff. There were even two papers on jointly modeling segmentation and tagging for Chinese at this year’s ACL (Jiang et al. and Zhang et al.). Joint modeling of this kind seems like a promising approach to allowing “intelligent” tokenization; by extending the tokenization model far enough, we could even maintain high end-to-end recall, which is not possible with a state-of-the-art first-best probabilistic tokenizer.