$1M Netflix Prize Qualifier BellKor Pragmatic Chaos

by

Finally! The BellKor Pragmatic Chaos team, winner of the last two Netflix Progress Prizes just topped the 10% improvement on the development set required to be evaluated for the grand prize of US$ 1M. Check out the the Netflix Prize Leaderboard.

Given what the public’s said about this contest before, I can’t say I’m too shocked at the public’s comments to the NY Times’ Big Blog post And the Winner of the $1 Million Netflix Prize (Probably) Is …. They were overwhelmingly mocking and negative. They thought the academics were giving Netflix free work. They thought 10% better than lousy recommendations were still lousy.

I’ve said this before, but what the public doesn’t realize is just how valuable this kind of data is. They’d probably be horrified at how much time we “waste” on bakeoffs like this one where there is no reward. A lot of groups out there probably would’ve paid Netflix for this kind of data, just like we pay the partly publicly funded Linguistic Data Consortium.

Plain and simple, this is a relatively huge data regression problem with data missing nonrandomly. The top teams achieved a fairly amazing reduction in variance by pooling multiple predictors. Given the (root) mean-square error (MSE) evaluation metric, and the relation of the bias-variance decomposition of mean square error, it’s not surprising the best systems were all ensembles of classifiers.

The effort required to fit these models and dream up structures and parameterizations was a Herculean effort. Check out the details up to last year in:

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s