I’d been using AUC of the P-R curve as the point statistic for comparison, as most of the algorithms predict conservatively and don’t actually reach 99% recall…

Okay, I’ll punt the raw results to you when we have clearance to share it. Thanks — always glad to have a new angle, and I don’t think Bayesianly very much myself.

What’s a “GS”?

I’m in no hurry; I know the biologists are close with their data and wouldn’t want to step on anyone’s toes.

1. Not everyone uses a probabilistic system,

2. Many IR systems don’t have an estimation/training phase, and

3. When running a retrospective evaluation, you typically just have ranked responses to queries.

Having said all that, it occurs to me from Aleks’s suggestions you might be able to use a bootstrap analysis.

Now, if you insist on doing significance on ROC curves, why not do a bunch of 2-fold cross validations over a number of permutations of the indices, and paint the ROC for each one of them? If you run 100 cross validations, you will have 100 ROC curves, and if you try two models on the same split, you will be even able to compare them. I’ve done such things in my PhD.

A bit of background on the system: It’s not actually a document IR system, it’s a set of algorithms (and ensembles) for selecting and ranking relevant proteins from a much larger set of candidates, using various evidence sources — gene expression profiles, evolutionary relatedness via domain fusion, MEDLINE co-occurrence, semantic similarity of GO annotations, yadda yadda.

Dave’s comment about the ‘traditional sciences’ is bang on, as this is going to a bioinformatics journal (formerly a trad molecular biology one), and yes, there is a certain element of ass-covering. However the deadline looms so the significance testing will probably have to wait until a reviewer requests it.

In the meantime, it’s just become an interesting intellectual challenge.

Ken, you’re right about the difference between significantly different vs. better, I was thinking of the former really. I think any reviewer who insisted on the latter would be making too many assumptions about what ‘better’ means, since as you point out, curves cross, and part of our motivation for presenting various ensembles is that different ones emphasize precision or recall.

Bob, can you do any informative hacking just based on the scores, or would you need the actual GS and predictions? If you need the raw data, I’ll need to ‘anonymize’ it, as our collaborators who assembled the GS by hand don’t want it published until their own paper is out.

They’re different questions because one curve can be higher on the left than another curve and lower on the right. The two curves are then different, but it’s a subjective question whether one is better than the other.

The first question also seems easier, because you “just” want to model whether the two curves might plausibly have been generated from the same system.

For the second question I’d probably want to integrate the PR function times a utility function that tells you the desired tradeoff between P & R. As Dave says though, carrying a significance test through that process seems horribly messy.