As a researcher, you sometimes has to answer questions like “how accurate is this particular test in detecting a specific disorder” or “how sensitive is this (imaging) device to  reveal a certain condition like breast cancer” or “how good is this test to predict later success, for example in higher education”. Like so many questions in science and psychology, the answer is no that simple.

A theory, called “signal detection theory” (SDT) can help. One of the pioneers, J.A. Swets, has published a very well written article in Scientific American. An unabridged and more difficult version can be downloaded [full text]  from  psychologicalscience.org. You can also find some visual explanations on the website anaesthetist.com. The following Excel workbook gives you the possibility to experiment  with the dice-game example from the handbook. You can also download the PASW dataset. Maybe, you want to use the Web-based Calculator for ROC Curves to calculate and draw the ROC curves.

Suppose that you do the following experiment. A group of children is presented with a  list of words and instructed to memorize them.  After that, they receive a second list with old words (previously presented) and new words intermingled (not previously showed but related). For each word they have to indicate how confident they are that the particular word is an OLD word on a 5-point rating scale (1 – Definitely negative to 5 – Definitely positive) and to make a response (OLD or NEW). Each child receives five lists. Some fictitious cases are presented in the following table (click to enlarge or download Excel-file).

In the first list, the subject 1 recognizes the first word correctly as an old (previously showed) word and is rather confident about it. The second word is also correctly identified as a new word but the child has doubts and is rather negative that is an old word. With the third word, the child makes a mistake and falsely recognizes a new word as an old one. The child is also confident that it is an old word. How accurate is this child in remembering?

In terms of signal detection, you can distinguish 4 situations. The correct responses are given by the true-positives and true-negatives; the incorrect responses by the false-positives and false-negatives.

What about a subject that has a high hit rate (true-positive probability). In the first list of the example, the child has a hit rate of 100% (4/4); every old word that is presented is recognized as such. Does this subject remember the words accurately? On first thought, our answer should be “yes”. The subject remembers the old words in all cases. This is quite a good performance.  The subject, however, recognizes also 1 new word as old (1/6 = 17%).The hit rate is very high but the false alarm rate is also substantial. In fact, a high hit rate can be obtained very easily by saying most of the time “OLD”, regardless if you remember or not the actual word. This is quite the opposite of a good performance.

In fact, several combinations are possible. They are visualized by a ROC-graph (Receiver Operating Characteristic). The X-axis represents the false-positives probability. The Y-axis shows the true-positives probability. The data points are:

FPP       TPP
0,000    0,000
0,040    0,400
0,080    0,760
0,200    0,880
0,560    0,960
1,000    1,000

Suppose that our subject only wants to respond OLD when his confidence rating is more than 5. He will have zero true-positives (word = old and response = old) and zero false-positives (word = new and response = old), simply because he never reponds OLD.  This is the first data point. Suppose that his cut-off value or criterion is 5  How many true-positives will he have (all five lists)? The combination word status= 1 and confidence = 5 appears 10 times on a total of 25 OLD words. This is a true-positive probability of 0.4. The false-positive probability is 0.04 (=1/25; third word in list 1). This is the second data point in our ROC-curve.

In our example the subject has 20 out of 25 times recognized the old word (TPP=0.8) and has 5 times responded old when in fact it was a new word (FPP=0.2). So, our subject has an implicit cut-off value of 3. When his confidence rate was 3, 4, 5 or more he responded OLD, creating 20% false alarms and 80% hits.