Section 2.3 Learning and evaluation

Thresholds, errors and calibration

In a constructed set of 1,000 cases, 50 are urgent. A system that classifies every case as ordinary achieves 95% accuracy while detecting zero urgent cases. Choosing a threshold calls for reading all four cells of the confusion matrix.

PolicyTrue positivesFalse positivesFalse negativesTrue negatives
A, lower threshold406410886
B, higher threshold302020930

Understand each denominator

For A, recall is 40/50 = 80%. Precision is 40/104 ≈ 38.5%. For B, both are 60%. Recall looks at cases known to be urgent. Precision looks at cases flagged by the system. If a missed urgent case costs eight units and an unnecessary review costs one, A costs 144 units and B costs 180 on this set. A capacity limit of 60 reviews would change the decision. The counts and costs are constructed to make the calculation inspectable.

Confusion matrix for policy A. The urgent row contains 40 and 10. The ordinary row contains 64 and 886.
Figure 2.3. Policy A flags 104 cases. Recall and precision divide the same numerator by two distinct populations.

Does a score of 0.8 really mean 80%?

A score can rank cases without providing a reliable probability. Calibration compares the observed event frequency within groups of scores to the stated probability. If roughly eight of ten requests scored near 0.8 are actually urgent, this group is consistent with calibration. A small group yields an uncertain estimate. Fit a calibration method on validation data kept separate from the test set, then check the effect of a change in the event’s base rate.