Comments on: To Stem or Not to Stem? http://lingpipe-blog.com/2007/03/21/to-stem-or-not-to-stem/ Natural Language Processing and Text Analytics Sat, 04 Feb 2012 20:56:48 +0000 hourly 1 http://wordpress.com/ By: lingpipe http://lingpipe-blog.com/2007/03/21/to-stem-or-not-to-stem/#comment-5731 Mon, 26 Oct 2009 19:55:26 +0000 http://lingpipe.wordpress.com/2007/03/21/to-stem-or-not-to-stem/#comment-5731 Indeed — I love Church’s paper. It’s nice to see someone try to measure the naivete of naive Bayes. And I think correlation is a good measure; it’s one that’s picked up by things like SVD approaches.

You may be right about the downside. I’m getting very frustrated by what seems like overstemming on Google’s part (I use an acronym that means one thing, they expand it into something else or vice-versa).

The other thing your comment made me realize is that there’s no way to recover from overstemming. You can recover from understemming by adding in disjunctions or prefix queries.

]]> By: Mike Schultz http://lingpipe-blog.com/2007/03/21/to-stem-or-not-to-stem/#comment-5715 Fri, 23 Oct 2009 23:53:38 +0000 http://lingpipe.wordpress.com/2007/03/21/to-stem-or-not-to-stem/#comment-5715 There’s a problem comparing search performance of stem no-stem systems to each other using aggregate, i.e. average measures. Those results tend to say, oh, it’s a wash, for some queries, stemming is better, for some no stemming is better. So let’s stem. The problem is, users perceptions weigh larger in the calculations, than an average measure can capture. The downside of over-stemming is never made up by the upside. Once a user sees “Amenities” -> “Amen”, “Heine” -> “Hein”, “Amy” -> “Ami”, “productivity” -> “produce” or the other bazillion wretched side effects of the porter stemmer, you have lost their trust. I think the prudent dividing line is the inflection/derivational distinction. People expect plural conflation to singlular, but Noun->Adjective->Verb->Noun? Please, that was a bad idea 30 years ago when it was born.

A great article on the efficacy of stemming is “One Term or Two?” from Ken Church. The way to motivate stemming is by being convinced that two terms should be represented by one in the first place. If not, then not.

]]>