Andrew Gelman and David Madigan wrote a paper on why 0-1 loss is so problematic:
- Gelman and Madigan. 2015. How is Ethics Like Logistic Regression? Chance 28(12).
This is related to the issue of whether one should be training on an artificial gold standard. Suppose we have a bunch of annotators and we don’t have perfect agreement on items. What do we do? Well, in practice, machine learning evals tend to either (1) throw away the examples without agreement (e.g., the RTE evals, some biocreative named entity evals, etc.), or (2) go with the majority label (everything else I know of). Either way, we are throwing away a huge amount of information by reducing the label to artificial certainty. You can see this pretty easily with simulations, and Raykar et al. showed it with real data.
Yet 0/1 corpora and evaluation remain the gold standard (pun intended) in most machine learning evals. Kaggle has gone largely to log loss, but even that’s very flat around the middle of the range, as Andrew discusses in this blog post:
- Gelman. 2016. TOP SECRET: Newly declassified documents on evaluating models based on predictive accuracy. Blog post.
The problem is that it’s very hard to train a model that’s well calibrated if you reduce the data to an artificial gold standard. If you don’t know what I mean by calibration, check out this paper:
- Gneiting, Balabdaoui, and Raftery. 2007. Probabilistic forecasts, calibration and sharpness. J. R. Statist. Soc. B 69.
It’s one of my favorites. Once you’ve understood it, it’s hard not to think of evaluating models in terms of calibration and sharpness.
and outcome vector 
. In regression modeling, we assume a coefficient vector 
and explicitly model the outcome likelihood 
and coefficient prior 
. But what about 
, where 
are the additional parameters involved in modeling 
. 
. 
.
.
others than a joint distribution on all 
variables.
for data 
, usually factored as the product of a likelihood and prior term, 
. Given some observed data 
of the the unknown parameter vector 
to carry out Bayesian inference. Note that the posterior is a whole density function—we’re not just after a point estimate as in maximum likelihood estimation.
parameterized by some new parameters 
. The mean-field form of variational inference factors the approximating density 
by component of 
, as
. 
until we see how they’re used in the variational inference algorithm. 
for the parameters ![\phi^* = \mbox{arg min}_{\phi} \ \mbox{KL}[ g(\theta|\phi) \ || \ p(\theta|y) ]](https://s0.wp.com/latex.php?latex=%5Cphi%5E%2A+%3D+%5Cmbox%7Barg+min%7D_%7B%5Cphi%7D+%5C+%5Cmbox%7BKL%7D%5B+g%28%5Ctheta%7C%5Cphi%29+%5C+%7C%7C+%5C+p%28%5Ctheta%7Cy%29+%5D&bg=ffffff&fg=333333&s=0 "\phi^* = \mbox{arg min}_{\phi} \ \mbox{KL}[ g(\theta|\phi) \ || \ p(\theta|y) ]")
.
![\mbox{{} \ \ set } \phi_j \mbox{ such that } g_j(\theta_j|\phi_j) = \mathbb{E}_{g_{-j}}[\log p(\theta|y)]](https://s0.wp.com/latex.php?latex=%5Cmbox%7B%7B%7D+%5C+%5C+set+%7D+%5Cphi_j+%5Cmbox%7B+such+that+%7D+g_j%28%5Ctheta_j%7C%5Cphi_j%29+%3D+%5Cmathbb%7BE%7D_%7Bg_%7B-j%7D%7D%5B%5Clog+p%28%5Ctheta%7Cy%29%5D&bg=ffffff&fg=333333&s=0 "\mbox{{} \ \ set } \phi_j \mbox{ such that } g_j(\theta_j|\phi_j) = \mathbb{E}_{g_{-j}}[\log p(\theta|y)]")
.
returning a single non-negative value, defined by![\mathbb{E}_{g_{-j}}[\log p(\theta|y)]](https://s0.wp.com/latex.php?latex=%5Cmathbb%7BE%7D_%7Bg_%7B-j%7D%7D%5B%5Clog+p%28%5Ctheta%7Cy%29%5D&bg=ffffff&fg=333333&s=0 "\mathbb{E}_{g_{-j}}[\log p(\theta|y)]")
![\begin{array}{l} \mbox{ } = \int_{\theta_1} \ldots \int_{\theta_{j-1}} \int_{\theta_{j+1}} \ldots \int_{\theta_J} \\[8pt] \mbox{ } \hspace*{0.4in} g(\theta_1|\phi_1) \times \cdots \times g(\theta_{j-1}|\phi_{j-1}) \times g(\theta_{j+1}|\phi_{j+1}) \times \cdots \times g(\theta_J|\phi_J) \times \log p(\theta|y) \\[8pt] \mbox{ } \hspace*{0.2in} d\theta_J \cdots d\theta_{j+1} \ d\theta_{j-1} \cdots d\theta_1 \end{array}](https://s0.wp.com/latex.php?latex=%5Cbegin%7Barray%7D%7Bl%7D++%5Cmbox%7B+%7D+%3D+%5Cint_%7B%5Ctheta_1%7D+%5Cldots+%5Cint_%7B%5Ctheta_%7Bj-1%7D%7D+%5Cint_%7B%5Ctheta_%7Bj%2B1%7D%7D+%5Cldots+%5Cint_%7B%5Ctheta_J%7D++%5C%5C%5B8pt%5D+%5Cmbox%7B+%7D+%5Chspace%2A%7B0.4in%7D++g%28%5Ctheta_1%7C%5Cphi_1%29+%5Ctimes+%5Ccdots+%5Ctimes+g%28%5Ctheta_%7Bj-1%7D%7C%5Cphi_%7Bj-1%7D%29+%5Ctimes++g%28%5Ctheta_%7Bj%2B1%7D%7C%5Cphi_%7Bj%2B1%7D%29+%5Ctimes+%5Ccdots+%5Ctimes+g%28%5Ctheta_J%7C%5Cphi_J%29++%5Ctimes++%5Clog+p%28%5Ctheta%7Cy%29++%5C%5C%5B8pt%5D+%5Cmbox%7B+%7D+%5Chspace%2A%7B0.2in%7D++d%5Ctheta_J+%5Ccdots+d%5Ctheta_%7Bj%2B1%7D+%5C+d%5Ctheta_%7Bj-1%7D+%5Ccdots+d%5Ctheta_1++%5Cend%7Barray%7D&bg=ffffff&fg=333333&s=0 "\begin{array}{l} \mbox{ } = \int_{\theta_1} \ldots \int_{\theta_{j-1}} \int_{\theta_{j+1}} \ldots \int_{\theta_J} \\[8pt] \mbox{ } \hspace*{0.4in} g(\theta_1|\phi_1) \times \cdots \times g(\theta_{j-1}|\phi_{j-1}) \times g(\theta_{j+1}|\phi_{j+1}) \times \cdots \times g(\theta_J|\phi_J) \times \log p(\theta|y) \\[8pt] \mbox{ } \hspace*{0.2in} d\theta_J \cdots d\theta_{j+1} \ d\theta_{j-1} \cdots d\theta_1 \end{array}")
independently.
such that ![g_j(\theta_j|\phi_j) = \mathbb{E}_{g_{-j}}[\log p(\theta|y)]](https://s0.wp.com/latex.php?latex=g_j%28%5Ctheta_j%7C%5Cphi_j%29+%3D+%5Cmathbb%7BE%7D_%7Bg_%7B-j%7D%7D%5B%5Clog+p%28%5Ctheta%7Cy%29%5D&bg=ffffff&fg=333333&s=0 "g_j(\theta_j|\phi_j) = \mathbb{E}_{g_{-j}}[\log p(\theta|y)]")
. Finding such approximating terms 
given a posterior 