Should I Build or Borrow a Text Annotation GUI?

by

There’s been a hot thread on the BioNLP mailing list about which text annotation GUI to use. In this case, they wanted to mark up relations. There were, of course, lots of tools suggested.

But I happen to agree with Peter Corbett, who said:

One option is to make your own. This is the option I took, and I’m very happy with the results. I spent a while looking at other annotation tools and finding them to be missing key features or cumbersome to use or difficult to translate the results into the right format. After discussing this in a group meeting, the consensus was that it would be better and easier to build our own tool than to build a pre- and post-processing pipeline to adapt our formats to a different annotation tool.

Of course, this only works if you have the resources in both time and programmer knowledge to do this. Luckily, we do here at Alias-i, so that’s what we’ve done in the following cases.

Named-Entity Annotation

I built a token-centric annotation tool that could be driven by the keyboard and by custom callback behaviors for dragging and highlighting. We’ve used this for annotating standard named entities and bibliographic citations and I can annotate around 5K tokens/hour accurately using it.

You can get this one from:

Alignment

I built a tool for aligning parallel sequences of symbols back when I was at Bell Labs. I was working on pronunciation models, so the symbols were short n-grams of chars and short n-grams of phonemes. I had to build some custom control behavior to get the aligned symbol slider interface right.

Morphology and Stemming

This was a project for a customer who wanted to develop corpora in multiple languages. I built a pretty simple unsupervised model (like Wicentowski’s supervised model), and used these annotations to train it.

Gene Linkage

This is still an active project. As part of our NIH grant, we’re linking gene mentions in MEDLINE to Entrez-Gene. Breck and Mitzi built this one to be served by Tomcat over the web and record results in a database. It uses our high-recall mention finder to highlight potential mentions, and then uses some heuristic matching to pull out possible matches from Entrez-Gene, OMIM and organize the results using Homologene.

Context-Free Grammar Trees

My favorite annotation tool was also built at Bell Labs. I used the inside-outside algorithm to derive probabilities of subtrees, and used a gradient green-yellow-red color scheme to indicate which were likely. The user clicked accept/reject on a node, and that was filtered from the set of possibilities, forward-backward was recalculated, and the new best analysis consistent with all previous clicks was put up.

We were using it for short sentences arising from a spoken dialogue system for movie listings. I don’t think it’d have scaled (as a GUI) to the 40+ word sentences common in newswire.

Tag a Little, Learn a Little

In all cases, I used the tag-a-little, learn-a-little paradigm, which first crossed my radar in the (apprently now retired) Alembic Workbench. I also auto-fed examples with a big “done” button that also allows you to page back and forth between examples to inspect and correct your previous work. I can’t stress just how important this is.

In all cases, we need to adjudicate results, but I’ve not built that into any of the interfaces.

Mechanical Turk

We’ve been using more and more mechanical turk interfaces. So far, I’ve done much simpler web-form based interfaces for named-entity, and Emily’s gone through three iterations until we found one that works for stemming. She’s also working on rolling out gene linkage to the Turkers this week. For named entity and stemming, their annotations are amazingly accurate, especially in aggregate if you give them pre-qualifying tests to reinforce the anno standard.

Excel or SQL Dumps

This is actually what all of our customers tend to do. This is particularly easy for classification and spell checking evaluation tasks.

5 Responses to “Should I Build or Borrow a Text Annotation GUI?”

  1. Rich W Says:

    It was odd to find me appearing in your blog postings…. :)

  2. lingpipe Says:

    In case other readers didn’t put “Rich W” together with my reference to “Wicentowski” in the text, here’s Richard Wicentowski’s home page. The system to which I was referring is described in this paper, which I’d highly recommend:

    R. Wicentowski. 2004. Multilingual Noise-Robust Supervised Morphological Analysis using the WordFrame Model. In Proceedings of Seventh Meeting of the ACL Special Interest Group on Computational Phonology (SIGPHON).

  3. Brendan O'Connor Says:

    wow, that’s quite a list for someone who claims they don’t want to “get into the data business”.. :)

  4. Hayri Volkan Agun Says:

    You should mention TrEd. It is a complex annotation tool that support hierarchical NE annotation.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s