Classification: Sensitivity, specificity, precision, recall

A score, a threshold, and four outcomes

Almost no classifier answers yes or no directly. It computes a score: a biomarker level in the blood, a relevance score for a document, or more generally a predicted probability. It then compares the score with a threshold. Anything above the line is flagged; anything below is let pass.

The figure shows the scores of two populations: the sick patients in orange, the healthy patients in blue. The two distributions overlap, so no threshold can separate them perfectly. Wherever you put the line, some orange lands on the wrong side, and some blue does too.

Directly under the curves, the four shaded areas are tallied for a population of 1,000 and arranged in a two-by-two grid. The rows are what is actually true; the columns are what the classifier said. That grid is the confusion matrix, so named because it shows which classes the classifier confuses with which. Three sliders drive it all at once: the threshold, the separation between the curves, and the prevalence, the share of patients who are actually sick. Tap a cell to see where it lives in the distribution.

The four cells are the four possible outcomes:

• True positive (TP): sick and flagged. The test caught the disease.
• False negative (FN): sick but not flagged. A miss.
• False positive (FP): healthy but flagged. A false alarm.
• True negative (TN): healthy and not flagged. A correct all-clear.

The colour code holds for the rest of the page. Hue shows what something truly is: orange for positive, blue for negative. Strength shows what the classifier did with it: strong for flagged, pale for let pass. Each cell of the table wears the colour of its area under the curves.

Each slider moves the table in its own way. The threshold trades one error for the other: slide it left and you catch more orange, but you also flag more blue. Counts move sideways within each row, from one column to the other. The separation is the only slider that reduces both errors at once, and it stands for a better classifier, not a better setting of this one. The prevalence leaves the curves untouched, because each is scaled to its own class, but it rescales the rows of the table.

That points to a structure worth noticing before any ratio is computed. The row totals are set by the world: how many patients are actually sick is a fact about the population, and the classifier cannot change it. The column totals are set by the classifier: how many cases get flagged depends on where the threshold sits. The tiny switch under the table relabels it for a search engine instead of a blood test: same numbers, different words.

Reading across, reading down

Every metric is a fraction, with some cells on top and some cells below. The four famous ones split cleanly into two kinds, depending on whether the denominator is a row or a column. Pick one to see which cells it uses: the numerator gets a solid ring, the rest of the denominator a dashed one, and the unused cells grey out.

Recall is sensitivity. Both are TP / (TP + FN). Library scientists asked what share of the relevant documents a search found; physicians asked what share of the sick a test detected. Both questions read the same row of the same table.

Specificity is not precision. Specificity reads the bottom row, TN / (TN + FP). Precision reads the left column, TP / (TP + FP). The two share a single cell, FP, and nothing else. A classifier can have 99% specificity and 5% precision at the same time, as the section on base rates will show.

Notice also which cell never appears in precision or recall: TN. In a library of millions of documents, the irrelevant ones you rightly did not retrieve are astronomically many, and nobody counts them. Precision and recall are the part of the table you can still measure when the bottom-right cell is unknowable. That is no accident of history. It is the reason the two vocabularies exist, as the history below explains.

The same table as overlapping sets

Retrieval researchers have always thought in sets. There is the set of relevant documents and the set of retrieved documents, and the goal is to make the two coincide. The figure draws them as circles inside a box. The box is the whole population, and each area is proportional to its count. The orange circle holds everything actually positive. The dashed circle holds everything the classifier flagged. Their overlap is TP. The parts of each circle outside the other are the two kinds of error, and everything outside both circles is TN. Pick a ratio to see which regions it uses: solid regions are the numerator, hatched regions complete the denominator, and everything else greys out.

Recall measures how much of the orange circle the dashed circle covers. Precision measures how much of the dashed circle lies inside the orange one. A perfect classifier makes the two circles coincide. Lowering the threshold inflates the dashed circle: it covers more of the orange, so recall goes up, but it also spills further into the blue, so precision goes down. Specificity is the odd one out. It lives almost entirely outside both circles, and it is the only one of the four that needs the box at all.

The F₁ score, a popular single-number summary of precision and recall, has a simple picture here. It is the Dice coefficient of the two sets: 2·|A ∩ B| / (|A| + |B|), the overlap measured against the average size of the two circles. It is high only when the circles are about the same size and overlap heavily.

Four ratios side by side

Here are the four famous ratios as pictures. Two of them are identical, and that is not a bug.

Solid regions are the numerator; hatched regions complete the denominator.

Why one table has so many names

The confusion matrix was not invented once. It was found at least four times, between the late 1920s and the mid-1950s, by fields that rarely read each other’s journals. Each started from a different practical question, and each question picks out a different pair of ratios from the same table.

Statistics asked about errors. Jerzy Neyman and Egon Pearson framed a hypothesis test as a choice between two hypotheses with two ways to be wrong. Rejecting a true null is a type I error, with rate α. Keeping a false one is a type II error, with rate β. The power of a test, 1 − β, is the chance of detecting an effect that is really there: sensitivity under another name. A statistician fixes α, the false positive rate, in advance and then maximises power. That amounts to choosing a point on what would later be called the ROC curve.

Medicine asked about the test. In 1947 Jacob Yerushalmy, a statistician with the U.S. Public Health Service’s tuberculosis program, had to compare four ways of reading chest X-rays. He defined sensitivity as the probability that a diseased person is called positive and specificity as the probability that a healthy person is called negative. Both condition on the truth, and that was deliberate. Someone evaluating a test wants a property of the test that travels unchanged from one hospital to the next. The patient’s question, “I tested positive, so am I ill?”, had to wait until 1966 for its name. That year Thomas Vecchio wrote about the predictive value of a test and warned that it depends on how common the disease is.

Radar asked about the operator. A radar operator watching a noisy screen had to decide whether a blip was an aircraft. Their vocabulary survives in signal detection theory: hit, miss, false alarm and correct rejection. Peterson, Birdsall and Fox formalised the theory at the University of Michigan in 1954, and Tanner and Swets carried it into psychology the same year. The central insight is that performance has two independent parts. One is how far the signal stands out from the noise (d′). The other is where the observer sets the cut-off (the criterion, our threshold). Plotting the hit rate against the false-alarm rate as the criterion moves gives the receiver operating characteristic, a radar engineer’s name that stuck. A false friend lurks here: in signal detection, “sensitivity” means d′, the separation of the curves, not the hit rate. Lee Lusted brought the ROC into radiology around 1960, and medicine has used it to compare tests ever since.

Libraries asked about the search. Mechanised literature search raised a new problem: how do you grade a search over a collection too large for anyone to read? Perry, Kent and Berry proposed in 1955 to judge a search by two fractions: how much of what was relevant it found, and how much of what it found was relevant. Cyril Cleverdon’s Cranfield experiments made these the recall ratio and the precision ratio. The Cranfield report of 1966 also lists the rival measures then in circulation: Perry’s “noise factor” (the complement of precision), a “fallout ratio” (the false positive rate), and, at Western Reserve University, a measure called “specificity”, defined exactly as Yerushalmy had defined it. So the librarians had the whole table, and they chose precision and recall anyway. In a collection of millions, the true negatives are enormous and uninformative. A search that wrongly returns 10 of a million irrelevant documents and one that wrongly returns 100 both have specificity above 99.98%, yet their precision can differ almost tenfold. John Swets himself argued in 1963 that retrieval should be judged by ROC curves. The field kept precision and recall, and Keith van Rijsbergen later folded the two into a single effectiveness measure whose complement is today’s F-score.

Machine learning borrowed from everyone. Kent Spackman imported ROC analysis into machine learning in 1989, and Tom Fawcett’s 2006 tutorial made it standard. Information extraction and natural-language processing brought precision, recall and F₁ over from retrieval. Biostatistics brought sensitivity and specificity. One classifier now serves all of these applications, so its papers use whichever dialect the application speaks. The vocabularies also collide now and then. An influential 1996 comparison of gene-finding programs by Burset and Guigó defined “specificity” as TP / (TP + FP), which is precision in everyone else’s vocabulary. The reason was the same as in the library: counting true negatives across a genome is hopeless.

Timeline

1. 1928–1933statistics

   Neyman & Pearson recast significance testing as a decision between two hypotheses, with two error types and the power of a test.

   coined: type I error (α) = FPR · type II error (β) = FNR · power = sensitivity
2. 1940sradar

   Radar operators in the Second World War must call blips on noisy screens. Their performance becomes an engineering problem, the “operating characteristic” of a receiver.

   vocabulary: hit · miss · false alarm · correct rejection
3. 1947medicine

   Jacob Yerushalmy compares X-ray techniques for tuberculosis screening and introduces two terms for a test’s accuracy.

   coined: sensitivity · specificity
4. 1950medicine

   W. J. Youden proposes a single index for rating diagnostic tests: sensitivity + specificity − 1.

   coined: Youden’s J
5. 1954radar & psychology

   Peterson, Birdsall & Fox publish the theory of signal detectability; Tanner & Swets apply it to human vision. Detection splits into discriminability and criterion.

   coined: ROC curve · d′ · hit rate / false-alarm rate
6. 1955retrieval

   Perry, Kent & Berry, working on machine literature searching, propose judging a search by the share of relevant documents found and the share of found documents that are relevant.

   origin of: recall · precision
7. 1960medicine

   Lee Lusted brings signal detection and the ROC curve into radiology and medical decision-making.

   import: ROC → medicine
8. 1963retrieval

   John Swets argues in Science that information retrieval systems should be evaluated with signal detection theory and ROC curves. The field mostly declines.

   attempted import: ROC → retrieval
9. 1966retrieval

   Cleverdon & Keen’s Cranfield report standardises retrieval evaluation and catalogues the competing names of the day.

   named: recall ratio · precision ratio · fallout ratio (= FPR) · Perry’s noise factor (= 1 − precision) · Western Reserve’s specificity
10. 1966medicine

    Thomas Vecchio writes on the predictive value of a single test in unselected populations and shows how it falls with prevalence.

    coined: positive / negative predictive value (PPV = precision)
11. 1975 · 1982statistics

    Bamber shows the area under the ROC curve equals the Mann–Whitney statistic; Hanley & McNeil popularise this in radiology. AUC becomes a probability: that a random positive outranks a random negative.

    named: AUC · AUROC
12. 1979retrieval

    C. J. van Rijsbergen’s Information Retrieval defines an effectiveness measure E combining precision and recall; F = 1 − E.

    origin of: F-measure · F₁ · F_β
13. 1982medicine

    David Eddy documents physicians confusing a test’s sensitivity with its predictive value in a mammography problem. The base-rate fallacy enters clinical teaching.

    named: confusion of the inverse
14. 1989machine learning

    Kent Spackman proposes ROC analysis for evaluating learned classifiers.

    import: ROC → machine learning
15. 1995statistics

    Benjamini & Hochberg introduce false-discovery-rate control for multiple testing: the expected share of rejections that are false, a column-wise ratio.

    coined: false discovery rate (FDR = 1 − precision)
16. 1996bioinformatics

    Burset & Guigó evaluate gene-finding programs and call TP/(TP + FP) “specificity”, because true negatives in a genome are uncountable.

    collision: “specificity” used for precision
17. 2006machine learning

    Tom Fawcett’s tutorial makes ROC analysis standard in machine learning. Davis & Goadrich prove that a curve dominates in ROC space exactly when it dominates in precision–recall space.

    bridge: ROC ⇔ PR
18. 2007medicine

    Gigerenzer and colleagues report that only about one in five of 160 gynaecologists picked the right answer to a mammography question, and show that natural frequencies fix most of the confusion.

    remedy: natural frequencies
19. 2015machine learning

    Saito & Rehmsmeier show that on imbalanced data the precision–recall plot is more informative than the ROC plot.

    advice: PR for rare positives

A Rosetta stone

Six ratios, and what each tradition calls them. The glyph shows which cells enter each ratio: solid cells are on top, outlined cells complete the bottom.

The pattern is easy to see. Fields that evaluate an instrument (a test, a receiver, a statistical procedure) condition on the truth and read across rows, because they want numbers that do not depend on who happens to walk in. Fields that evaluate an output (a result list, a set of discoveries) condition on the decision and read down columns, because the user only ever sees what was flagged. Both are right, and they answer different questions.

Base rates: when a good test is mostly wrong

Sensitivity and specificity are properties of the test. Precision is a property of the test and the population. The difference matters most when the condition is rare.

Take the classic version, in Gigerenzer’s numbers. About 1% of the women screened have breast cancer. A mammogram detects 90% of cancers (sensitivity) and falsely flags 9% of healthy women (false positive rate, so specificity is 91%). A woman tests positive. What is the chance that she has cancer? In Gigerenzer’s 2007 survey, most gynaecologists overestimated it badly. The answer is about one in ten.

Natural frequencies make the answer obvious, and that was Gigerenzer’s point. Out of 1,000 women, 10 have cancer and 9 of them test positive. Of the 990 who do not, 89 test positive too. That gives 98 positives, 9 of them real. Each orange square above is a real positive, and each strong blue square is a false alarm.

The chart below follows four ratios as the prevalence varies, holding the test fixed. Drag across it. The two flat lines are sensitivity and specificity, which read across rows and so ignore prevalence entirely. Precision collapses toward zero as the condition becomes rare, while the negative predictive value climbs toward 100%.

At the radar screen

The ROC curve was born at screens like this one. A plan position indicator (PPI) shows a radar’s surroundings from above. The antenna turns, a bright sweep line turns with it, and every echo it receives paints a spot of glowing phosphor at the matching bearing and range. Aircraft return strong echoes. The receiver’s own noise returns weak ones, everywhere, all the time, and the ground right around the antenna adds a little clutter.

The simulated scope below divides its picture into 360 bearings × 96 range cells: 34,560 cells per turn of the antenna. In each cell the receiver sees noise, or noise plus an echo. An automatic detector marks every cell whose return exceeds a threshold with an amber square. Tap a square to interrogate it, the radar version of asking “friend or foe?”, and learn whether anything is really there. Switch on reveal truth to see every real contact.

The tally under the scope is a confusion matrix built up scan by scan. Its rows count real contacts and empty cells; its columns count what the detector did. The labels are the radar operator’s words for the four cells: hit, miss, false alarm, correct rejection.

Watch the bottom row. At a threshold of 4σ, an empty cell raises a false alarm only about three times in 100,000. But a scan has 34,000 empty cells, so that still means roughly one false plot per turn. At 3σ it is nearly fifty per turn, far more than there are aircraft. The false alarm rate is tiny; the number of false alarms is not. So the precision of a plot, the chance that an amber square is a real contact, collapses. This is the base-rate problem again, in its original setting. It also explains why nobody looks at the bottom-right cell, with its hundreds of thousands of correct rejections.

That is why real radars set their threshold the way Neyman and Pearson set α. Choose an acceptable false-alarm rate first, then take whatever detection probability the echo strength allows. Detectors that keep the false-alarm rate fixed as the noise level changes are called constant false alarm rate (CFAR) processors. They matter wherever the noise floor varies: rain, ground returns or sea waves near the radar raise it, and a fixed threshold would fill the inner rings with false plots.

The ROC curve: every threshold at once

Every threshold produces its own confusion matrix. Sweep the threshold from very high to very low and record, at each step, two row-wise rates: the true positive rate (sensitivity) and the false positive rate (1 − specificity). The path they trace is the ROC curve. The radar engineers drew it to describe a receiver; here it describes the classifier as a whole, separate from any one threshold.

Drag horizontally across the plot to move along the curve. The small distribution strip above it moves in step.

The corners are easy to read. The point (0, 0) flags nothing, (1, 1) flags everything, and (0, 1) is a perfect classifier. The dashed diagonal is a coin flip: a classifier that flags at random, at any rate, catches positives and negatives in equal proportion. The better the classifier, the further its curve bows toward the top-left corner. Push the separation to zero and the curve collapses onto the diagonal.

Youden’s J is the vertical distance from the diagonal to the current point, TPR − FPR, which equals sensitivity + specificity − 1. Its maximum is a common default threshold. It is the right choice exactly when a missed positive costs as much as a false alarm once each is weighted by its prevalence, and that is often not the case.

Notice what does not move the ROC curve: prevalence. Both axes read across rows, so the ROC describes the classifier, not the population. That makes it ideal for comparing tests across settings, and blind to the base-rate problem.

Checking the AUC by drawing pairs

Draw one sick and one healthy patient at random and compare their scores. Each line joins a pair. A solid line means the positive scored higher, so the pair is ranked correctly; a dashed line means it is misranked. The share of solid lines converges to the AUC.

The precision–recall curve

The precision–recall curve sweeps the same threshold but plots precision against recall. It is the retrieval tradition’s curve, and it never uses TN. Below, the ROC curve (left, or above on a phone) and the PR curve show the same classifier at the same threshold. Now move the prevalence slider.

The ROC curve does not move. The PR curve sinks as positives become rare. Its baseline, the dashed line, is the prevalence itself, because a classifier flagging at random has precision equal to the base rate at every recall. The faint arcs are iso-F₁ contours: every point on an arc has the same F₁, the harmonic mean of precision and recall.

Davis and Goadrich (2006) proved that one classifier dominates another in ROC space exactly when it dominates in PR space. In that sense the two curves hold the same information about how well the scores rank. They weight it very differently, though. When positives are rare, the ROC curve’s x-axis divides by an enormous number of negatives, so even a flood of false alarms barely moves the point. The interesting region gets squeezed into a sliver near zero, and the AUC looks reassuring. The PR curve shows the same flood as collapsing precision. If what you care about is the quality of what gets flagged, plot PR.

The whole table, every name

Here is every ratio that commonly gets a name, computed live from the current table. A ✓ marks the ratios that do not depend on prevalence, and they are exactly the ones built from rows alone. A glyph framed by rows or columns marks a composite of row-wise or column-wise rates.

Which number should you report?

• Report the confusion matrix itself, along with the prevalence. Every ratio can be recomputed from it, and none of the ratios can recover it.
• To compare classifiers or tests apart from where they will be used, use row-wise measures: sensitivity and specificity, the ROC curve, and the AUC.
• To deploy where positives are rare and every flag costs attention (fraud queues, search results, screening follow-ups), use column-wise measures: precision at your operating threshold and the PR curve.
• Distrust accuracy under imbalance. At 1% prevalence, a classifier that never flags anything is 99% accurate. Balanced accuracy, Youden’s J or the Matthews correlation don’t fall for this.
• Single numbers hide the trade-off. The AUC averages over thresholds you will never use. F₁ ignores TN entirely and weights precision and recall equally. Choose the threshold from the real costs of each kind of error, then report what happens at that threshold.

Sources & further reading

• J. Neyman & E. S. Pearson, “On the problem of the most efficient tests of statistical hypotheses”, Phil. Trans. R. Soc. A 231, 1933.
• J. Yerushalmy, “Statistical problems in assessing methods of medical diagnosis, with special reference to X-ray techniques”, Public Health Reports 62(40), 1947, 1432–1449.
• W. J. Youden, “Index for rating diagnostic tests”, Cancer 3, 1950, 32–35.
• W. W. Peterson, T. G. Birdsall & W. C. Fox, “The theory of signal detectability”, IRE Trans. Information Theory 4, 1954; W. P. Tanner & J. A. Swets, “A decision-making theory of visual detection”, Psychological Review 61, 1954.
• J. W. Perry, A. Kent & M. M. Berry, “Machine literature searching X”, American Documentation 6(4), 1955, 242–254.
• L. B. Lusted, “Logical analysis in roentgen diagnosis”, Radiology 74, 1960.
• J. A. Swets, “Information retrieval systems”, Science 141, 1963, 245–250.
• C. W. Cleverdon & M. Keen, Factors Determining the Performance of Indexing Systems, vol. 2, Cranfield, 1966: the page listing “fallout”, the “noise factor” and Western Reserve’s “specificity”.
• T. J. Vecchio, “Predictive value of a single diagnostic test in unselected populations”, NEJM 274, 1966, 1171–1173.
• D. Bamber, “The area above the ordinal dominance graph and the area below the receiver operating characteristic graph”, J. Math. Psychology 12, 1975; J. A. Hanley & B. J. McNeil, “The meaning and use of the area under a ROC curve”, Radiology 143, 1982.
• C. J. van Rijsbergen, Information Retrieval, 2nd ed., Butterworths, 1979, ch. 7.
• D. M. Eddy, “Probabilistic reasoning in clinical medicine”, in Kahneman, Slovic & Tversky (eds.), Judgment under Uncertainty, 1982.
• K. A. Spackman, “Signal detection theory: valuable tools for evaluating inductive learning”, Proc. 6th Int. Workshop on Machine Learning, 1989.
• Y. Benjamini & Y. Hochberg, “Controlling the false discovery rate”, JRSS B 57, 1995.
• M. Burset & R. Guigó, “Evaluation of gene structure prediction programs”, Genomics 34, 1996.
• T. Fawcett, “An introduction to ROC analysis”, Pattern Recognition Letters 27, 2006; J. Davis & M. Goadrich, “The relationship between precision–recall and ROC curves”, ICML 2006.
• G. Gigerenzer et al., “Helping doctors and patients make sense of health statistics”, Psychological Science in the Public Interest 8, 2007.
• T. Saito & M. Rehmsmeier, “The precision–recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets”, PLOS ONE 10, 2015.