“Sentiment Takes a LONG Time”, Said the Logistic Regression Classifier

by

Hello again, fellow LingPipers!  It is I, yet again, Sam The Intern.  In my last (and first) post I promised the masses that I would return with a study similar to the last one that I conducted, but this time doing the non-trivial case of sentiment.  It turns out that this aspiration ended up being nothing more than a (Ling)Pipe-dream.  As my titular character, the logistic regression classifier so concisely said before, sentiment takes a long time!  And, with this being my last week at LingPipe, I will not have enough time to give our logistic regression classifiers the attention and whispered sweet nothings that they require to function properly.

As you may remember from my first post, I absolutely love to annotate.  However, annotating for sentiment takes much more time than language identification, because the annotator needs to internalize and judge the text, not just glance at it.  Add to this the fact that the sentiment classifier requires much more training data than the language identification classifier, and you have yourself an intern that has been product-conditioned to Coca-Cola, but no time to do anything about it (reading 1000 tweets about how much people love coke will do that to you).

I was able to start the study, with small amounts of success.  The following confusion matrix is the result of  a 10-fold cross-validation run across all of the data that I had time to annotate (about 1000 tweets each for “Clinton”, “Coke”, and “Gaga”).  The top of the chart is the reference data which I annotated, and the side is the response data that the classifier provided.  The categories are: w = neutral, e = positive, q = negative.

   w    e   q
w 1244,239,50
e 313 ,312,23
q 199 ,45 ,23

I wish that I had more time to spend on the sentiment classification problem.  Perhaps I will be back next summer, to continue the effort to teach computers good from bad.  Until then, I will be at Rochester, college-ing!  And drinking massive quantities of Coke…

One Response to ““Sentiment Takes a LONG Time”, Said the Logistic Regression Classifier”

  1. Bob Carpenter Says:

    Thanks for dumping out the confusion matrix. Moms aside, I think this is the easiest way to understand classifier behavior.

    Performance on negatives is particularly bad. of the 100 or so negative items, only 25% were classified correctly — that’s actually worse than chance. If you threw balanced darts, you’d expect 1/3 to be correct.

    Although not as large a problem with logistic regression as naive Bayes, you still have a problem with hugely imbalanced data. With only 1/30 tweets being negative and the classifiers being trained on per-item metrics (log loss), it’s not too surprising this is what happened. Oddly, though, the system returned 175 or so negative answers, so false positives on negatives are particularly high.

    As I was saying at the office, you can often take classifiers that are this bad and still get useful work out of them.

    One, you might be able to cherry pick the high-confidence classifications (scores close to 1.0) and do a reasonable job of finding negative and positive tweets.

    Two, you can adjust for the biases in various ways (see the section in the LingPipe book on classifiers) so that you can estimate overall prevalence of positive and negative tweets.

    Given that positive terms are likely to be different for Clinton (is this mostly Hillary now or did Bill and George get in there, too?), Gaga (senile person, big fan of something, or pop start) and Coke (drug, beverage or precursor to steel). The problem with keyword searches is it doesn’t distinguish them.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Follow

Get every new post delivered to your Inbox.

Join 823 other followers