Comments on: Russ Altman on “Poisoning Your Prior”

By: Clair Branhan

Clair Branhan — Mon, 03 Oct 2011 18:35:42 +0000

Hi Sandie. So fare i can see, Arbuckle has answered your question. I dont see any sense.

By: lingpipe

lingpipe — Wed, 22 Jul 2009 18:43:51 +0000

I see the semantic confusion — what you call “validation” is what I’d call “testing” or “evaluation”, not what I’d call “development”. During development, the prior’s in play for tuning. During evaluation, it really can’t be, because you could get any result you wanted.

This brings up an issue with reporting cross-validation results rather than true held-out test results. I think of cross-validation like development, not like testing. The problem often found in papers is a confusion of optimal cross-validated parameters with true test results. It’s too easy to overfit cross-validation, so despite the name, tuned cross-validation results aren’t to be trusted for what Russ is calling “validation”.

Of course, the real evaluation we care about is prediction of things we don’t yet know. Too bad that’s so much work in biology.

By: Russ Altman

Russ Altman — Wed, 22 Jul 2009 17:59:36 +0000

As you know, for the purposes of academic publication, one needs to prove the performance of any new classifier and a common way to do this is to emulate how it will do in the world when it is let loose on real, unsolved problems. In that case, we save some known stuff that is not used in training, so we can see how the algorithm would do on these “gold standard” known results, as a proxy for how it will do on true things that we don’t know later. That’s what I mean by the validation phase.

By: lingpipe

lingpipe — Wed, 22 Jul 2009 17:54:28 +0000

Russ: Thanks for the response. I figured I might be misinterpreting something. Is there a paper with the statistical details? Were you talking about the M-BISON paper on micro-array association? It looks like a very interesting use of informative priors. I was trying to emphasize that from a Bayesian perspective, the prior is just another parameter in the model, and shouldn't get any special treatment. I see model-building as a continuous loop of tuning and testing fit, so I'm confused by what you mean by "validation phase". Do you mean looking to see that an interaction you expect to see is actually found by the algorithm? With fine-grained enough priors you could percolate instances to the top, which would definitely fall into the realm of overfitting the model.

By: Russ

Russ — Tue, 21 Jul 2009 23:05:20 +0000

NO! You misinterpreted what I was saying. I said the quote to stress that you can’t build the answer into your prior *during the validation phase*. In order to have a credible validation, you need to set your prior, seal it, and THEN apply the method to make predictions. Once your method is validated and you are making real predictions. go crazy and use everything you know–but during validation, don’t cheat!