Because ERR only depends on editorial grade, if you can predict the editorial grade of a document/query vector pair, you can optimize the ranking.

I’d guess Yahoo! was just playing it safe on privacy after the AOL and Netflix debacles.

The problem with treating the learning to rank problem as a mere classification or regression problem is that you don’t use the *relative* position of documents with each other, and they are more general problems to solve.

Something worth mentioning is that queries in the test set don’t exist in the training set. This means that the feature vectors necessarily contain information about both the document d and the query q, not d alone.

I agree that the absence of the raw data is a bit disappointing. I guess two possible explanations are Yahoo not willing to disclose this data and data size. This means that the learning and possibly feature selection/dimensionality reduction are the main things that will distinguish the teams.

Interestingly, the current 20 best teams are all in a 0.01 range, with regards to the ERR.

It’s really hard for us poor (in the cash sense, not the “woe is me” sense) startups to interpret all the legalese. My own eyes sort of glaze over at the language, which always seems vaguely menacing.

And about the clause 4a: I’m not a lawyer, but my understanding is that this clause is meant to prevent an entanglement resulting from simultaneous participation in two challenges with conflicting rules.