Signal Detection Theory

An important model of decision making that has been largely ignored in JDM research goes by the name of signal detection theory (SDT). The name reflects the roots of the model, which was in guiding early radar operators in deciding whether a given display included a "signal" (e.g., a real target) hidden in the "noise" on the radar screen. The SDT approach is driven by practical prescriptive goals of improving decision making and is only indirectly concerned with the psychology of the decision maker. The approach is, however, of great generality for many applied problems, from assessing cracks in aircraft wings to detecting breast cancer, and from evaluating job candidates to testing for AIDS, drug use, or lying.

SDT (Getty, Pickett, D'Orsi, & Swets, 1988; Swets, 1988) considers a diagnostic situation, one in which repetitive choices must be made between two alternatives. An evidence system of some kind produces probabilistic information of imperfect accuracy to guide the choices. For example, a test for some specific disease might produce a numerical score: If the score is high, the patient is likely to have the disease; if it is low, he or she is unlikely to have it. What should you, the physician, do with a given score? Because the test is imper­fect, there is a possibility of an error either way. If you act as though the disease is present when it is not (a false positive), you incur costs of wasteful, painful, and perhaps dangerous treatments and patient distress. If you act as though the dis­ease is absent when it is actually present (false negative), you incur costs of failing to treat real disease. You need to set a threshold on the test score at which you will act. The thresh­old requires consideration of how likely the disease is to start with (the base rate) and of the costs and benefits of the two different sorts of error you might make.

The evidence system offers the decision maker a set of choices, which can be summarized in a plot of false-positive probabilities versus true-positive probabilities, called a re­ceived operating characteristic (ROC) another echo of SDT s

Probability of a False Positive Figure 19.3 Diagnostic systems in signal detection theory.

roots in radar curve (Figure 19.3). The decision maker may decide to set a very strict threshold, insisting on a very high test score so that the chance of a false positive is small. The price she pays is that she will miss many true positives. Using the same system, she could choose a lax threshold, act­ing even when test scores were quite low. Doing this would push the true-positive probability higher, but only at the cost of more false alarms. The ROC curve is thus a summary of the evidence system's accuracy. A highly accurate system would offer very high true-positive probabilities with small false-positive probabilities. A completely useless system would offer identical probabilities of each. Anything that pushes the ROC up and to the left (higher true-positive prob­ability for the same false-positive probabilities) represents an improvement in accuracy and offers the decision maker a better range of options at which to set the threshold. Curve A thus offers a better menu of choices than does Curve B, and one research goal is to improve existing diagnostic systems in this way.

Independently of this improvement, it is possible to help the decision maker set appropriate thresholds so as to make the best choice from those offered by the ROC curve. (Con­sider, e.g., if you would want to use the same threshold on an HIV test for screening blood donations and for evaluating real patients. A false positive on the first case merely wastes a pint of good blood. In the second case, it would erroneously lead a patient to believe that he or she had a life-threatening disease.)

An excellent example of the SDT approach is given in Getty et al. (1988), in which the problem is improving the di­agnosis of malignant breast cancers from mammograms. The authors were able to devise a checklist and scoring system of features that the radiologists were to score from each image,

and this led to significant improvement in the accuracy f the evidence system (ROC curve). The enhanced procedur offered real improvements over the existing methods. F_0f 1,000 cases with a cancer prevalence of 32% (the population that their study addressed), they estimated that the improved procedure would identify an additional 42 malignancies (with no additional false positives), 82 fewer false-positives (with no additional missed malignancies), or various blends in be­tween. Given the seriousness of the disease, and of both sorts of error, the enhancement offered by the SDT analysis is clearly significant.

Date: 2016-03-03; view: 905