Diagnostic Precision from Sensitivity, Specificity and Prevalence

by

Suppose you have a diagnostic for your favorite binary condition (YFBC). For the sake of concreteness, let’s consider the mammogram test for breast cancer. The American College of Preventive Medicine provides an estimate of between 0.75 and 0.90 sensitivity and 0.9 to 0.95 specificity. Let’s give the mammogram the benefit of the doubt and assume the high end, with sensitivity of 0.90 and specificity of 0.95.

Sensitivity

Recall that sensitivity is the accuracy of the test on women who have breast cancer. In other words, if a woman who has breast cancer gets a mammorgram, the test will come back positive 90% of the time because the test has a 0.9 sensitivity. Cases where the condition is present and the test is positive are called true positives (TP). For 10% of the women with breast cancer, the mammogram test result will be a false negative (FN), where the test misdiagnoses a woman who has the disease.

Specificity

Specificity is the accuracy of the mammogram for women who do not have breast cancer. That is, if a woman does not have breast cancer, the test will be negative 95% of the time. If a woman does not have breast cancer and the test results are negative, the outcome is called a true negative (TN). The other 5% of the time we get a false positive, which means the test is positive for a woman who does not have breast cancer.

Prevalence

Before we can calculate diagnostic precision, we need to know the prevalence of the condition, which is the percentage of the population that has the condition. We rarely know population prevalences, but they can be estimated. For breast cancer, the following paper provides estimates of prevalence of breast cancer, which increase with age:

Consulting the chart above, the prevalence of breast cancer is roughly 0.015 for women aged 50.

Precision (aka Positive Predictive Value)

The precision of a test, also known as its positive predictive value, is the chance that a subject testing positive actually has the condition being tested for. Precision is thus defined as TP/(TP+FP).

My Mammogram is Positive, Do I Have Breast Cancer?

Now let’s suppose you’re a 50-year old woman whose mammogram was positive. What’s the chance you have breast cancer? Let’s calculate all the probabilities.

  • p(TP) = prevalence * sensitivity = 0.015 * 0.9 = 0.014
  • p(FN) = prevalence * (1 – sensitivity) = 0.0015
  • p(TN) = (1 – prevalence) * specificity = 0.97 * 0.95 = 0.94
  • p(FP) = (1 – prevalence) * (1 – specificty) = 0.049

Thus a generous estimate (remember we took the upper bounds for the accuracy estimates) of the chance of a 50-year old having cancer given that she tests positive on a mammogram is only:

p(TP)/(p(TP)+p(FP))=0.21

Now consider a 40-year old woman getting a mammogram. The prevalence is only .005 (less than half a percent), leading to a test precision of 0.083, meaning only 1 in 12 40-year old women with a positive mammogram will actually have breast cancer.

If the sensitivity and specificity are on the low end of the estimate, namely sensitivity is 0.7 and specificity 0.90, precision for 50-year old subjects is only .096 (less than 1 in 10), and that for 40-year old subjects 0.033, or about 1/30.

That’s why you only start getting diagnostic tests when you’re older — they’re just too inaccurate for the young and likely healthy. Or if you’re known to be more likely to get breast cancer, for instance because of incidence in the family or genetic tests; the prevalence for your situation will determine the test’s predictive accuracy.

The U.S. National Cancer Institute recommends “Women in their 40s and older should get a mammogram every 1 to 2 years.” That’s a whole lot of chances for false positives.

Low positive predictive accuracy is also why there are follow-up imaging tests (sonograms) even before biopsies. But keep in mind that the same caveats apply to those imperfect tests.

More Information

For more information on other tests (sonogram, MRI, physical exam and various biopsy procedures), do a PubMed Search. This article from the National Cancer Institute looks interesting:

In a survey of 50+ facilities between 1996 and 2002, they found varying values for sensitivity and specificity among facilities and by procedure (specialist in breast cancer, reading, double versus single reading, etc.), and an overall positive predictive value of mammograms of only 4% (meaning 1/25 women who test positive actually have breast cancer).

4 Responses to “Diagnostic Precision from Sensitivity, Specificity and Prevalence”

  1. Ken Says:

    This is fine, provided the threshold for a positive is chosen (if possible) to give sensitivity and specificity that result in sensible outcomes. Missing breast cancer has serious consequences but a false positive simply results in a biopsy which can be easily done, so it is not a problem. This is something that people are researching in other areas, the consequences of each decision, but it requires decent data on what happens in each of the four possible outcomes.

    One problem is that many diagnostic tests are designed to be used after some other symptom is found, not as screening tools. A headache without other explanation and with certain symptoms deserves a scan, and anything found can be a possible cause. No symptoms and anything found is probably unimportant, but now there is a dilemma, should they have a look or just wait and see what happens. Not a lot of data on this but neurosurgery can have severe consequences so nobody should do anything unless there is an obvious defect.

  2. lingpipe Says:

    Great points. Indeed, you often only get tests when there is some reason to believe you fall in a high-prevalence group. Much of the epidemiology literature’s concerned with multiple tests, which requires estimating correlation among the tests to derive sensible final estimates.

    Mammograms are screening tests, at least in the U.S. if you’re following NCI’s advice. Even simple needle biopsies are not completely without infection risk and local anesthetic risk. Of course, surgical biopsies are more dangerous, but they also have higher sensitivity. The major risk seems to be false positives resulting in needless major surgery; I have no idea how big a risk that is for breast biopsies.

  3. lingpipe Says:

    Why am I not surprised by a new set of recommendations from the United States Preventive Services Task Force. Other groups are sticking with a recommendation of 40. There’s more info in:

    In the calculations above, we were being fairly generous with diagnostic accuracy estimates.

    These numbers need to be constantly updated due to changes in sensitivity and specificity of the test(s).

    In a fuller model, the exact age at which it makes sense to be tested will also vary based on predictive features of the patients, such as genetic profiles. These would change the prevalence in the above calculations, and thus the received diagnostic precision for patients with a given set of predictors (e.g. age, genetic profile, health history).

  4. Owen M. Says:

    Starting a site much like this one got me to get into some research and I found your post to be extremely helpful. My site is centered around the idea curing cancer by stopping the angiogenic process. I wish you good luck with your research in the future and you can be sure I’ll be following it.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s