RESUMEN
Predictor variables for multivariate rules are frequently selected by methods that maximize likelihood rather than information. We compared the discrimination and reproducibility of a prediction rule for pneumonia derived using extended dependency analysis (EDA), an information maximizing variable selection program, with that of a validated rule derived using logistic regression. Discrimination was measured by receiver-operating characteristic (ROC) analysis, and reproducibility by rederivation of the rule on 200 replicate samples of size 250 and 500, generated from a training cohort of 905 patients using Monte Carlo techniques. Four of the five predictor variables selected by EDA were identical to those selected by logistic regression. With each variable weighted by its conditional contribution to total information transmission, EDA discriminated pneumonia and nonpneumonia in the training cohort with an ROC area of 0.800 (vs 0.816 for logistic regression, p = 0.60), and in the validation cohort with an area of 0.822 (vs 0.821 for logistic regression, p = 0.98). EDA demonstrated reproducibility comparable to that of logistic regression according to most criteria for replicability. Replicate EDA models showed good discrimination in the training and testing cohorts, and met statistical criteria for validation (no significant difference in ROC areas at a one-tailed alpha level of 0.05) in 80.8% to 94.2% of cases. We conclude that extended dependency analysis selected the most important variables for predicting pneumonia, based on a validated logistic regression model. The information-theoretic model showed good discriminatory power, and demonstrated reproducibility according to clinically reasonable criteria. Information-theoretic variable selection by extended dependency analysis appears to be a reasonable basis for developing clinical prediction rules.
Asunto(s)
Teoría de la Información , Modelos Logísticos , Aplicaciones de la Informática Médica , Computación en Informática Médica , Análisis Multivariante , Neumonía/epidemiología , Estudios de Cohortes , Humanos , Oportunidad Relativa , Neumonía/etiología , Reproducibilidad de los Resultados , Factores de RiesgoRESUMEN
It has been suggested that clinical prediction rules are not reproducible, and that the most important variables frequently do not appear in replicate models. The authors studied the reproducibility of a validated rule for predicting radiographic evidence of pneumonia (ROC areas for the training and validation cohorts, 0.816 and 0.821, respectively). Two hundred replicate samples of size 250 and size 500 were generated by sampling without replacement from the original training cohort of 905 patients with a 14.6% prevalence of pneumonia. Forward selection was performed among 31 candidate variables by stepwise logistic regression. Using as reproducibility criteria: 1) inclusion of all five variables from the original model in the original order; 2) inclusion of all five variables in any order; 3) inclusion of the first three variables; 4) inclusion of the first two variables; 5) inclusion of the first variable; and 6) inclusion of any of the five variables: 2.5%, 13.5%, 48.5%, 85.5%, 98.0%, and 100% of replicate models of sample size 500, respectively, met the criteria, whereas 0%, 0%, 16.5%, 49.0%, 71.5%, and 97.5% of models of sample size 250 met the criteria (all comparisons by sample size p < .0001 except for criteria 1 and 6, p = 0.07). Mean ROC areas in the training and validation samples were 0.829 and 0.791 for replicate models of sample size 500, and 0.831 and 0.779 for models of sample size 250. There was no significant difference in ROC areas between training and validation cohorts for 80.5% of models of sample size 500, and for 75.3% of models of sample size 250.(ABSTRACT TRUNCATED AT 250 WORDS)