There’s been a fair bit written on tokenization in the bio-NLP world, because of the complex token structures of biochemistry nomenclature (e.g. “p53wt” might become three tokens and “mRNA” two tokens).

Another place you see it is in learning to segment unsegmented input (e.g. phonemes into words).

There’s also a fair bit of attention in IR to issues of compound splitting, because of the Spider-man/Batman problem (“Spider-man” is two words, “Batman” is one, but many searchers get this wrong).

One problem is that tokenization isn’t “natural”. Speakers of a language don’t have intuitions about tokenization, so no one ever runs bakeoffs on tokenization. So the “right” tokenization is often system dependent. Having said that, you could say the same about POS tagging.

A big issue is that much of the test data is pre-tokenized. Consider CoNLL, the Penn Treebank, the Google N-gram corpus, etc. etc. Bakeoff organizers often consider this a service (e.g. Senseval and many KDD-like efforts), and only provide stemmed, tokenized and stoplisted input.

What’s really frustrating is getting a resource without a tokenizer (e.g. CoNLL, Google N-grams). Then you can’t convert your system to run on wild text. At least the Penn Treebank distributes its crazy script (which first does sentence detection so it can treat end-of-sentence periods differently than sentence-internal ones).

Then there are the probabilistic tokenizers, like in BioIE. I have too much carryover from my programming language days, but I like my tokenizers to be standalone and deterministic preprocessors, not part of some big joint probabilistic inference system. That’s why we tend to run finer-grained tokenizers than most.

For our spelling correction and for our rescoring named entity detector, we rely heavily on tokenization. For spelling, we can determine the set of tokens which may be suggested as tokens and make correction sensitive to that. For rescoring named entities, we use a simple HMM, which determines hard, but fine-grained token boundaries, then rescore it in a non-tokenized longer-distance model.

I have seen people running around asking “Um… do you have a tokenizer?” at various institutions in more than one country, and many end up using some idiosyncratic Perl script that “somebody hacked up one night in the lab”.

Even for English, to date there is no standard component that people can share, there is not even interest in the problem. Yes, people will tell you their experiences, but they would never consider publishing on the topic, because they consider it “solved”.

Interesting comments. The beam size of the demo is 1, which is the size of our system we reported in the paper. We didn’t get much improvement when increasing the beam size to 100, but it was painfully slow. So everything we did was plain greedy.

Unfortunately, I don’t know anything about PHP, I’m using a standard stub in our group for creating demos, and is has many problems.

But giving it a second thought, I do agree that character encoding, dealing with punctuation etc are important. One thing that I didn’t talk in the paper, due to lack of space, but which had impact on the system performance for real-world text was tokenization and text normalization. We’re using two tokenization schemes in our system, which play with how we parse hyphens and punctuation marks… All this thing is not trivial and under-researched.

@Lev Thanks for the link correction; I fixed it in the post.

And if I didn’t make it clear enough in the post, I really like both your approach and how well it seems to work on the examples I tried (many more than I put in the post; those were just the first two I tried). I’d urge everyone else to try it. The demo’s very speedy, too, which is great. How big is the demo beam?

I’m assuming something’s going wrong on your program’s decoding side, because when I cut and paste from the NY Times, in Latin1, to your web form, also Latin1 (both with inferred encodings from firefox), the characters get mangled on output. Is there a way I could’ve entered the curly apostrophe with the form the way it is now?

I have no idea how to properly deal with different encodings in PHP. It’s a pain in Java servlets, because (a) you have to transcode through Latin1 to recover bytes, because the servlet interface is specified in unicode characters [or you have to do everything in bytes, losing the advantage of letting the implementation parse forms for you], and (b) you then have to either know the underlying charset (how I’ve built our demos) or determine it using something like a unicode library (how we deal with untrusted char data).

Then there’s the issue of what to do with these things in the models. You wind up with names and things like that with very low frequency characters in them that aren’t in any of the character-based training data. We’ve had to deal with issues like this in training models from the French Treebank, because live French uses all sorts of different punctuation and the training corpus was fairly uniform.

Thanks for evaluating our system. I have two comments.
1) The Link you provided to our system is a bit incorrect.
This is the link to use: http://l2r.cs.uiuc.edu/~cogcomp/LbjNer.php
2) I hope you’ll agree with me that as a non-commercial demo we have hard time adjusting to different encodings without our web demo. It’s really easy to fix the encoding prbolem, and I’m a bit disappointed you didn’t try our system AFTER fixing the encoding, which is a much more fair comparison. I’ve done it, and the output is below. As you can see, our system does much better on the blog entry you mention, which, I agree with you is hard, since it’s not a newswire text.

Scope out these new exclusive pics of [PER Michael Jackson ] with two of his three kids – [LOC Paris ] and [PER Michael Joseph Jr ] . ( also known as Prince [PER Michael ] ) – from the new issue of [LOC OK ] ! . In this picture , [PER Prince ] , 4 , and [PER Paris Jackson ] , 3 , play dress-up in 2001 at the [LOC Neverland Ranch ] in [LOC Santa Barbara County ] , [LOC Calif ] . Also pictured is [LOC Michaelat Neverland ] , celebrating [PER Prince ] ‘s sixth birthday in 2003 with a [MISC Spider-Man-themed ] party . Cute !