For simplicity, we’ll only consider binary classifiers, so the population consists of positive and negative instances, such as relevant and irrelevant documents for a search query, or patients who have or do not have a disease.
Quite often, the available training data for a task is not balanced with the same ratio of positive and negative instances as the population of interest. For instance, in many information retrieval evaluations, the training data is positive biased because it is data annoted from the top results of many search engines. In many epidemiological applications, there is also positive bias because a study selects from a high risk subpopulation of the actual population.
Even with unbalanced training data, we might still want to be able to calculate precision, recall, and similar statistics for the actual population. It’s easy if we know (or can estimate) the true percentage of positive instances in the population.
Specificity and Sensitivity vs. Precision and Recall
Using the usual notation for true and false positives and negatives,
Sensitivity and specificity are accuracies on positive () and negative cases (), respecitively.
Prevalence for a sample may be calculated from the true and false positive and negative counts, by
Recall is just sensitivity, but precision is the percentage of positive responses that are correct, namely
Prevalence Adjusted Contingency Matrix
Suppose we know the test population prevalence , the probability of a random population member being a positive instance. Then we can adjust evaluation statistics over a training population with any ratio of positive to negative examples by recomputing expected true and false positive and negative values.
The expected true and false positive and negative counts in test data with prevalence over samples, given test sensitivity and specificity and , are
Sensitivity and Specificity are Invariant to Prevalence
Although the adjusted contingency matrix counts (true and false positive and negatives) vary based on prevalence, sensitivity and specificity derived from them do not. That’s because specificity is accuracy on negative cases and sensitivity the accuracy on positive cases.
Plug-In Population Statistic Estimates
Precision, on the other hand, is not invariant to prevalence. But we can compute its expected value in a population with known prevalence using the adjusted contingency matrix counts,
The counts all cancel; we can really work with true and false positive and negative rates instead of counts.
This is related to my earlier blog post on estimating population prevalence using a classifier with known bias and an unbiased population sample:
The approach described in the linked blog post may be used to estimate the population prevalence given only an arbitrarily biased labeled training set and an unlabeled and unbiased sample from the population.