Comments on: Convexity of (Root) Mean Square Error, or Why Committees Won the Netflix Prize

By: Viviane

Viviane — Sat, 04 Feb 2012 20:56:48 +0000

Thank you Felipe. I have read your work. It is ineedd the first to my knowledge that uses the extended (also called Fisher’s non-central) hypergeometric distribution for IR. I found that sampling from extended hypergeometric becomes exponentially expensive as the sample size increases, and that there is ongoing research in statistics about .While researching on the use of the hypergeometric distribution for IR, I found a paper from Wilbur dating back to 1993! Wilbur models the vocabulary intersection between the query and a set of relevant documents using the central hypergeometric distribution. Little has been done since then, probably because the multinomial distribution is a good approximation to the hypergeometric for most IR scenarios, i.e., when the sample size (query) is cosiderably smaller than the population size (document). However, as we show in the paper, in the case of document-long queries, the multinomial approximation does not hold anymore, and the use of the “vanilla” hypergeometric distribution is required.

By: Search terms and the flu: prefering complex models | Ready-to-hand

Search terms and the flu: prefering complex models | Ready-to-hand — Tue, 01 Dec 2009 07:54:32 +0000

[...] One way of doing this is just taking a weighted average of the predictions of several simpler models. This works quite well when your measure of the value of your model is root mean squared error (RMSE)… [...]

By: lingpipe

lingpipe — Wed, 30 Sep 2009 16:11:02 +0000

@Max Good point about convexity and ranking. Non-convex error functions make optimization-based estimation (training) difficult, too.

I don’t think the kind of non-convexity you’d get from top N or F-measure like scores would necessarily mean committee voting wouldn’t work for these kinds of tasks.

I think precision-at-N, where “relevant” means a customer rents it, is a better measure for Netflix’s task.

Furthermore, I think we need more results diversity — I don’t want it to recommend seven seasons of the Simpsons even if I will eventually rent them all and rank them highly. And I don’t want to keep being shown the same movie again and again as a recommendation, so there needs to be history.

I would also like Netflix and Amazon and PubMed to go deeper into more-like-this, because I find it useful for exploration.

Netflix’s own Cinematch system, which provided the baseline, was also based on regression. From what Netflix has said, they’re already incorporating some of the techniques from the competition. Certainly I’d think the features would port.

By: Max Gubin

Max Gubin — Wed, 30 Sep 2009 05:40:16 +0000

Actually, the situation with a real recommendation system is more complex because it can provide only a limited number of recommendations for a user, in other words, it provides only N top items with the best scores for a user. Any loss function that uses such a top window is extremely non-convex with many local minima (like loss functions in IR: P@N, NDCG). Popular Netflix methods like SVD and stochastic gradient descent won’t work well there. I won’t be surprised if people run into this problem when they try applying Netflix competition results in real systems.