RESUMEN

RATIONALE AND OBJECTIVES: The authors performed this study to determine whether satisfaction of search (SOS) errors in patients with multiple traumas are caused by faulty visual scanning, faulty recognition, or faulty decision making. MATERIALS AND METHODS: A series of radiographs were obtained in patients with multiple traumas. Radiologists interpreted each series under two experimental conditions: when the first radiograph in the series included a fracture, and when it did not. In the first experiment, the initial radiographs showed nondisplaced fractures of the extremities (minor fractures); in the second experiment, the initial radiographs showed abnormalities of greater clinical importance (major fractures). Each series also included a radiograph with a subtle (test) fracture and a normal radiograph on which detection accuracy was measured. In each experiment, gaze dwell time was recorded as 10 radiologists reviewed images from 10 simulated cases of multiple trauma. RESULTS: An SOS effect could be demonstrated only in the second experiment. Analysis of dwell times showed that search on subsequent radiographs was shortened when the initial radiograph contained a fracture; however, the errors were not based on faulty scanning. CONCLUSION: The SOS effect in musculoskeletal trauma is not caused by faulty scanning. Demonstration of an SOS effect on test fractures with major but not minor additional fractures suggests that detection of other fractures is inversely related to the severity of the detected fracture.

Asunto(s)

RESUMEN

RATIONALE AND OBJECTIVES: The authors performed this study to determine whether defective pattern recognition or defective decision making is more to blame for satisfaction of search (SOS) errors in chest radiography. MATERIALS AND METHODS: Fifty-eight chest radiographs-half of which demonstrated diverse, native abnormalities-were read by 20 observers. The radiographs were read twice, once with and once without the addition of a simulated pulmonary nodule. Observers provided a verbal account of their focus of attention, indicating suspicious features and regions considered during their inspection of the radiograph. Observers also provided a separate account of the abnormalities they would include in a radiologic report. RESULTS: When the authors considered only those reports that did not refer to the simulated nodules, they found no reduction in the area under the proper receiver operating characteristic (ROC) curves in cases that contained nodules. A smaller SOS effect, however, was demonstrated with analysis of events in which the native abnormality was missed in one condition but not the other. Verbal protocols suggested that the SOS errors were mainly caused by recognition failure rather than faulty decision making. CONCLUSION: Describing their focus of attention may have prompted observers to inspect the radiographs in a more deliberate, systematic way, thus reducing the SOS effect. More residual SOS errors were caused by defective pattern recognition than by faulty decision making.

Asunto(s)

RESUMEN

RATIONALE AND OBJECTIVES: The purpose of this study was to test whether the satisfaction of search (SOS) effect in chest radiology could be demonstrated with proper receiver operating characteristic (ROC) curves and with joint detection and localization ROC curves. MATERIAL AND METHODS: Data from an earlier ROC study of SOS in chest radiology were analyzed with three proper ROC models and one ROC model for joint detection and localization. Fits of the models were compared on the basis of likelihood-ratio chi-squared statistics (G2). To examine further the validity of the SOS effect in chest radiology, the authors also replicated the earlier study with a new sample of readers, analyzing the new data with the same methods. RESULTS: The proper contaminated binormal model fit the data better than the other two proper ROC models. Contaminated binormal analysis of the earlier and the replication experiment demonstrated an SOS effect: a reduction in area under the ROC curve for detection of the native abnormalities with the addition of nodules. Similarly, joint ROC analysis producing curves that appropriately cross the chance line gave similar results. CONCLUSION: Preventing inappropriate chance line crossing reduces measurement error and provides more powerful statistical tests. Results of both experiments showed that the SOS effect in chest radiology can be demonstrated with ROC methods that avoid inappropriate crossing of the chance line.

Asunto(s)

RESUMEN

RATIONALE AND OBJECTIVES: Receiver operating characteristic (ROC) data with false-positive fractions of 0 are often difficult to fit with standard ROC methods and are sometimes discarded. Some extreme examples of such data were analyzed to evaluate the nature of these difficulties. MATERIALS AND METHODS: Rating reports of fracture for single-view ankle radiographs were analyzed with the binormal ROC model and with two ROC models that keep the ROC curve from crossing the chance line. Because fractures were almost never reported that were not present, some views and locations yielded only ROC points with false-positive fractions of 0, while others yielded at least one ROC point with a non-0 false-positive fraction. RESULTS: The models tended to yield ROC areas close to or equal to 1. ROC areas of 1 imply a true-positive fraction close to 1; yet the data contained no such fractions. When all false-positive fractions were 0, the true-positive fraction could be much higher for one view than another for all observers. ROC areas gave little or no hint of these unmistakable differences in performance. CONCLUSION: These data challenge the validity and robustness of current ROC models. A key aspect of ankle fractures is that some may be visible on one view but not at all visible on another.

Asunto(s)

RESUMEN

RATIONALE AND OBJECTIVES: A contaminated binormal receiver operating characteristic (ROC) model is proposed to account for ROC data that have very few false-positive reports even though many healthy subjects are sampled. The model assumes that no signal information is captured for a proportion of abnormalities, and that these abnormalities have the same distribution as noise along the latent decision axis. MATERIALS AND METHODS: The authors developed a formal psychophysical model, presented here in detail. They have specified the psychophysical assumptions of the theory, and have provided proofs that include all essential details, from assumptions to implications. With the technical details that are provided, this theory can be implemented with computer programs to fit data. RESULTS: The new model can fit ROC data in which some or all of the ROC points have false-positive fractions of 0 and true-positive fractions of less than 1, without implying that performance is perfect. The resulting ROC curves are always proper, never exhibiting inappropriate chance line crossings. The model predicts that, under certain conditions, a bimodal categorical rating histogram will be observed for the signal distribution. The model predicts a relationship between the mean and standard deviation of the signal distribution and holds that, for expert decision makers, there are situations in which the prevalence and utility matrix preclude operating points in some ROC regions. The model has a straightforward extension to the joint detection and localization ROC curve. CONCLUSION: The contaminated binormal model accounts for ROC data with few or no false-positive reports.

Asunto(s)

RESUMEN

RATIONALE AND OBJECTIVES: The authors' purpose was to evaluate how well the contaminated binormal receiver operating characteristic (ROC) model fits (a) degenerate data for which standard ROC models commonly fail and (b) nondegenerate data from exemplary experiments, for which the standard binormal model should be appropriate. MATERIALS AND METHODS: The authors studied two examples of binormally degenerate data, with and without interior points, and ROC rating data from four experiments in visual psychophysics and radiology. The plots of contaminated binormal ROC curves of the binormal degenerate data were examined. For ROC data with at least one interior point, the new model was compared with conventional models on the basis of likelihood-ratio chi2 statistics (G2). RESULTS: With no interior points, the contaminated binormal model gave results consistent with the fundamental principle underlying ROC analysis, that is, for a fixed false-positive probability, the higher the true-positive probability, the better the diagnostic performance. Contaminated binormal ROC curves go through the empirical ROC points of the degenerate data without crossing the chance line or climbing far above the true-positive fractions of the points. For several model ROC studies, the contaminated binormal model gave smaller G2 results than conventional ROC models, although the differences tended to be small, usually with little difference in ROC area. CONCLUSION: The contaminated binormal model fits binormal degenerate data better than conventional ROC models, and it offers an explanation for the degeneracy. The lower G2 values on some classic, nondegenerate ROC data suggest that contamination may not be limited to degenerate ROC data.

Asunto(s)

RESUMEN

OBJECTIVE: We wanted to determine whether the standard three-view ankle radiographic series could be replaced by a two-view combination, and if so, which two-view combination (anteroposterior with lateral or mortise with lateral) would be superior. MATERIALS AND METHODS: During a 12-month period, we retrospectively reviewed 556 consecutive ankle radiographic studies consisting of anteroposterior, mortise, and lateral views. One hundred twenty patients with at least one ankle fracture were paired with 140 healthy control subjects. Each image in the three-view examination was separated and sorted by view and studied independently; all images were reviewed by two skeletal radiologists and two orthopedic surgeons. Each radiograph was evaluated for fracture of the medial, lateral, and posterior malleoli and the foot using a five-point confidence rating. Performance of each view and modeled two- and three-view combinations of views was evaluated with modified receiver operating characteristic analysis. RESULTS: The data provide little support for preferring either two-view combination (anteroposterior-lateral or mortise-lateral) for any type of fracture. The three-view combination does detect significantly more fractures than some two-view combinations in some locations, and there is a statistically significant cost in diagnostic accuracy for eliminating the anteroposterior or mortise view. CONCLUSION: Reducing the ankle radiographic series from three to two views would result in a small but significant decrease in the detection of fractures of the ankle and foot. Both two-view combinations are equivalent for fracture detection.

Asunto(s)

RESUMEN

RATIONALE AND OBJECTIVES: The authors conducted a series of null-case Monte Carlo simulations to evaluate the Dorfman-Berbaum-Metz (DBM) method for comparing modalities with multireader receiver operating characteristic (ROC) discrete rating data. MATERIALS AND METHODS: Monte Carlo simulations were performed by using discrete ratings on fully crossed factorial designs with two modalities and three, five, and 10 hypothetical readers. The null hypothesis was true for all simulations. The population ROC areas, latent variable structures, case sample sizes, and normal/abnormal case sample ratios used in another study were used in these simulations. RESULTS: For equal allocation ratios and small (Az = 0.702) and moderate (Az = 0.855) ROC areas, the empirical type I error rate closely matched the nominal alpha level. For very large ROC areas (Az = 0.961), however, the empirical type I error rate was somewhat smaller than the nominal alpha level. This conservatism increased with decreasing case sample size and asymmetric normal/abnormal case allocation ratio. The empirical type I error rate was sometimes slightly larger than the nominal alpha level with many cases and few readers, where there was large residual, relatively small treatment-by-case interaction and relatively large treatment-by-reader interaction. CONCLUSION: The results suggest that the DBM method provides trustworthy alpha levels with discrete ratings when the ROC area is not too large and case and reader sample sizes are not too small. In other situations, the test tends to be somewhat conservative or slightly liberal.

Asunto(s)

RESUMEN

RATIONALE AND OBJECTIVES: The authors tested the hypothesis that satisfaction of search effect, which is associated with the failure to detect native chest abnormalities in the presence of simulated nodules, is caused by reduced gaze on the native abnormalities. MATERIALS AND METHODS: Gaze dwell time of 20 radiologists was recorded for the region around abnormalities on images. Ten radiographs were reviewed, nine of which contained native abnormalities. Each image was seen with and without a simulated nodule. RESULTS: The decrease in the rate of true-positive findings in the detection of native abnormalities on images that contained simulated nodules confirmed the occurrence of a satisfaction of search effect. Gaze times on native abnormalities (up to the time of report of the abnormalities) were the same for images with nodules in which native abnormalities were missed (gaze time, 9.4 seconds) as they were for images without nodules in which native abnormalities were detected (gaze time, 9.5 seconds). Gaze time on missed native abnormalities was not affected by the presence (7.80 seconds) or absence (7.45 seconds) of nodules. CONCLUSION: Reduction in gaze dwell time on the missed abnormalities is not the cause of satisfaction of search errors in chest radiographs.

Asunto(s)

RESUMEN

RATIONALE AND OBJECTIVES: The standard binormal model is the most commonly used model for fitting receiver operating characteristic rating data; however, it sometimes produces inappropriate fits that cross the chance line with degenerate data sets. The authors proposed and evaluated a proper constant-shape bigamma model to handle binormal degeneracy. METHODS: Monte Carlo samples were generated from both a standard binormal population model and a proper constant-shape bigamma model in a series of Monte Carlo studies. RESULTS: The results confirm that the standard binormal model is robust in large samples with no degenerate data sets and that the standard binormal model is not robust in small samples because of degenerate data sets. CONCLUSION: A proper constant-shape bigamma model seems to solve the problem of degeneracy without inappropriate chance line crossings. The bigamma fitting model outperformed the standard binormal fitting model in small samples and gave similar results in large samples.

Asunto(s)

RESUMEN

RATIONALE AND OBJECTIVES: Extraintestinal abnormalities visible without contrast material on abdominal radiographs are reported less frequently when contrast examinations are performed. Gaze dwell time was used to determine whether this difference is due to failure by observers to scan plain-film regions of contrast studies or discounting of plain-film abnormalities that were actually scanned. METHODS: Patients were included whose contrast studies had elicited the largest reductions in positive responses compared with their plain-film studies in a previous detection experiment. Gaze of 10 radiologists was studied. RESULTS: Significantly less time was spent gazing at non-contrast regions of contrast examinations than at the corresponding regions of radiographs. Errors with radiographs were based primarily on failures of recognition and decision making, whereas errors with contrast studies were based primarily on faulty scanning. CONCLUSION: Satisfaction of search errors on contrast examinations are caused by reduction in scanning of noncontrast regions.

Asunto(s)

RESUMEN

RATIONALE AND OBJECTIVES: Observer performance studies sometimes use too few cases for estimating diagnostic accuracy from binormal receiver operating characteristic (ROC) curves. One important problem is degenerate data sets. We compared a new algorithm, RSCORE4, with the exact-solution approach to degeneracy in ROCFIT and with the Wilcoxon statistic. METHODS: Degenerate ROC solutions result from empty cells in the data matrix. We addressed this problem by adding a small constant to empty cells in a maximum-likelihood program, RSCORE4. When this method failed, the program branched to a pattern-search algorithm. We tested the program in a series of Monte Carlo studies. RESULTS: RSCORE4 converged to nondegenerate solutions in every case and gave results closer to population values than ROCFIT or Wilcoxon. ROCFIT converged to exact-fit degenerate solutions, those with zero or infinite parameter values, in more than 40% of the samples. The Wilcoxon statistic was biased. CONCLUSION: RSCORE4 seems to outperform other currently recommended methods for dealing with degeneracy.

Asunto(s)

RESUMEN

RATIONALE AND OBJECTIVES: We tested whether having clinical information would improve perception or simply decision making. METHODS: Sixty-four pediatric chest and abdominal radiographs, half of which had abnormalities, were presented to nine radiologists under one of two conditions. In one condition, history consistent with abnormalities actually present for positive cases was provided for positive and matched negative cases before inspection. In a second condition, this information was provided only after inspection was completed and the radiograph was no longer available. Because detailed visual memory is short-lived, the image information was no longer available when the history was provided after inspection. A control condition measured detection without history. RESULTS: Detection was significantly better with history provided before inspection. Detection did not differ for history provided after inspection and inspection without history. CONCLUSION: The only difference between conditions with history was in whether history influenced perception; history affected decision making in both conditions. Clinical history affected perception in interpreting radiographs, not simply decision making.

Asunto(s)

RESUMEN

RATIONALE AND OBJECTIVES: Satisfaction of search (SOS) occurs when a lesion is "missed" after detecting another lesion in the same radiograph. The authors investigated the SOS effect in abdominal contrast studies. METHODS: The authors measured detection of 23 plain film abnormalities in 43 patients who had plain film and contrast examinations. Each plain-film and contrast study was examined independently by 10 radiologists in two sessions, with receiver operating characteristic (ROC) curve areas estimated with the computer program RSCORE-J (University of Iowa, Iowa City, IA) for each condition. RESULTS: Observers more often missed plain film abnormalities present on contrast studies but also made fewer false-positive (FP) responses. There was no change in ROC area, but decision criteria grew more conservative. CONCLUSIONS: The reduction of detecting plain-film abnormalities in contrast examinations differs from the SOS effect of other imaging studies. The reduction in true-positive (TP) and false-positive (FP) rates suggests that a different cause may underlie these misses.

Asunto(s)

RESUMEN

RATIONALE AND OBJECTIVES: Facilitation of detection by clinical history generally has been found with a single abnormality per image but not with multiple abnormalities. Multiple abnormalities per image can occasion a "satisfaction-of-search" effect in which detection of one lesion is reduced in the presence of other distant lesions. Our experiment studied the combined effect of multiple abnormalities and clinical history on accuracy. METHODS: Detection of native lesions was measured 1) with histories suggestive of the native abnormality; 2) with these histories and added simulated pulmonary nodules; and 3) with the same added nodules and histories suggestive of metastatic disease. These conditions also were compared with those of a previous experiment that were similar but included no history. RESULTS: Detection was substantially improved for appropriately prompted abnormalities even in the presence of a pulmonary nodule. In fact, satisfaction of search was not found in the presence of an appropriate history. Detection of unprompted abnormalities was unchanged when prompts indicated other abnormalities actually present. Prompted abnormalities were detected earlier in search. CONCLUSIONS: History appears to direct perceptual resources to the prompted abnormalities, thereby alleviating satisfaction of search. The presence of nodules yielded a small but consistent reduction in total search time for searches involving false responses, suggesting that satisfaction of search may depend more on reduction in search time than had been indicated by previous research.

Asunto(s)

Asunto(s)

RESUMEN

"Satisfaction of search" (SOS) refers to the effect in which a second lesion remains undetected after detection of another lesion on the same radiograph. The objective of this study was to clarify our understanding of SOS by relating it to total time of inspection and time intervals before, between, and after discovery of lesions. Detection accuracy of native lesions in chest radiographs, before and after the addition of a simulated nodular lesion, was measured for ten observers. Analysis of data from this and a previous experiment showed that average perceptual accuracy of individual receiver operating characteristic curves was significantly reduced with the addition of the nodules. Plots and analyses of search time revealed that, on average, during a typical 46-second inspection of a case, simulated nodules were found at 18 seconds, native abnormalities at 25 seconds, and false positives occurred at 33 seconds. Time needed to find nodules did not depend on whether native lesions were present; time to find native lesions did not change with addition of nodules; and total search time was the same for images with one, two, or no lesions. The detection results show that the SOS effect was obtained, but that interrupting search in order to measure it also diminishes accuracy. Analysis of the time course data relates SOS to perceptual capture and strategic halting of search.

Asunto(s)

RESUMEN

A subset of underreading errors (false-negative responses) in radiology has been attributed to "satisfaction of search," which occurs when lesions remain undetected after detection of an initial lesion. This phenomenon has not been studied in the experimental laboratory. A primary goal of this study was to develop a procedure or paradigm to study satisfaction of search. The authors measured detection accuracy for native lesions in images before and after the addition of a simulated nodular lesion. Simulated and native lesions were not spatially superimposed and the native abnormalities were physically identical with and without the nodules. Only responses related to the native lesion were analyzed. Accuracy parameters of receiver operating characteristic (ROC) curves were estimated by the method of maximum likelihood and jackknife. The average perceptual accuracy of the individual ROC curves as measured by Az and de' was significantly reduced with addition of the nodules (t = 2.364, p = 0.025, t = 2.648, p = 0.017, respectively). Az and de' parameters of the pooled ROC curve showed a similar effect (t = 1.573, p = 0.080; t = 1.934, p = 0.047, respectively). The results indicated a substantial satisfaction-of-search effect, with diminished accuracy in perception of native lesions.

Asunto(s)

Asunto(s)

RESUMEN

The authors reexamined a suspected difference in the effects of clinical information upon detection of nodules vs. more diverse lesions by directly incorporating classification specificity into detection ROC analysis. Categorical prompts, correct for specific abnormalities, led to detection superior to unprompted reading when non-nodule trials (various lesion types) were analyzed. Trials that contained pulmonary nodules, or contained no lesions but had been preceded by "possible tuberculosis" or "rule out metastatic disease" prompts, failed to demonstrate the diagnostic prompt superiority. Perceptual responses may differ when nodules are compared with more complex lesions.

Asunto(s)