Methods for Estimating Areas under Receiver-Operating Characteristic Curves: Illustration with Somatic-Cell Scores in Subclinical Intramammary Infections
Detilleux, Johann; Arendt, J.; Lomba, F.et al.
1999 • In Preventive Veterinary Medicine, 41 (2-3), p. 75-88
[en] The aim of this study was to demonstrate receiver-operating characteristic (ROC) methodology in the context of bovine intramammary infection (IMI). Quarter somatic cell scores (SCS) were available to evaluate quarter IMI, and the final IMI diagnosis was made from milk bacteriologic cultures. Data consisted of 11,453 quarter-milk samples collected on 2084 clinically healthy cows located in 154 Belgian herds. Bacteriological analyses showed 16.2%, 7.2%, and 11.9% of quarters infected with coagulase-positive Staphylococcus spp., Streptococcus agalactiae, and coagulase-negative Staphylococcus spp., respectively. The ROC curve indicated all the combinations of sensitivity and specificity that quarter SCS was able to provide as a test to identify quarter IMI. Among parametric, semi-parametric, and non-parametric methods to estimate area under ROC curves, the parametric method seemed the least appropriate for analyzing SCS in this study. With the non-parametric method, the total area under the ROC curves showed quarter SCS could identify quarter IMI with an overall accuracy of 69%, 76%, and 59% for coagulase-positive Staphylococcus spp., S. agalactiae, and coagulase-negative Staphylococcus spp., respectively. Parametric and non-parametric statistical tests showed that overall SCS diagnostic capability was significantly (p<0.01) different from chance and was different (p<0.01) across the three bacteria. However, the SCS thresholds yielding the highest percentage of quarters correctly classified as infected (for the observed prevalence and for equal costs assigned to false-positive and false-negative results) were so high that they had no practical value. The major advantage of ROC analysis is the comprehensive description of the discrimination capacity of SCS for all possible choices of critical values. The major disadvantage is the dependency upon the gold standard used for the final diagnosis--but recent improvements of the methodology will correct the problem.
Disciplines :
Agriculture & agronomy
Author, co-author :
Detilleux, Johann ; Université de Liège - ULiège > Département de productions animales > Génétique quantitative
Arendt, J.
Lomba, F.
Leroy, Pascal ; Université de Liège - ULiège > Département de productions animales > Biostatistique, économie, sélection animale
Language :
English
Title :
Methods for Estimating Areas under Receiver-Operating Characteristic Curves: Illustration with Somatic-Cell Scores in Subclinical Intramammary Infections
Arendt, J., Detilleux, J., Bughin, J., Leroy, P., Lomba, F., 1997. Usefulness of quarter somatic cell counts for detecting udder infections in dairy cows. Proceedings of a meeting of European Association for Animal Production, Prague.
Bamber D. The area above the ordinal dominance graph and the area below the receiver operating graph. J. Math. Psychol. 12:1975;387-415.
Coulon J.B., Dauver F., Garel J.-P. Facteurs de variation de la numération cellulaire du lait chez des vaches indemnes de mammites cliniques. INRA Prod. Anim. 9(2):1996;133-139.
Craven N., Williams M.R. Defenses of the bovine mammary gland against infection and prospects for their enhancement. Vet. Immunol. Immunopathol. 10:1985;71-127.
DeLong E.R., DeLong D.M., Clarke-Pearson D.L. Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics. 44:1988;837-845.
Dohoo I.R., Leslie K.E. Evaluation of changes in somatic cell counts as indicators of new intramammary infections. Prev. Vet. Med. 10:1991;225-237.
Dorfman D.D., Alf E. Maximum-likelihood estimation of parameters of signal detection theory and determination of confidence intervals - rating method data. J. Math. Psychol. 6:1969;487-496.
Erskine R.J., Elberhart R.J. Comparison of duplicate and single quarter milk samples for the identification of intramammary infections. JDS. 71:1988;584-586.
Hajian-Tilaki K.O., Hanley J.A., Joseph L., Collet J.-P. A comparison of parametric and nonparametric approaches to ROC analyses of quantitative diagnostic tests. Med. Decis. Making. 17:1997;94-402.
Hanley J.A., McNeil B.J. The meaning and use of area under a receiver operating characteristics (ROC) curve. Radiology. 143:1982;29-36.
Hollander, M., Wolfe, D.A., 1973. Nonparametric Statistical Methods. Wiley, New York, pp. 67-78.
Jones G.M., Pearson R.E., Clabaugh G.A., Heald C.W. Relationship between somatic cell counts and milk production. J. Dairy Sci. 67:1983;1823-1831.
Kairisto V., Poola A. Software for illustrative presentation of basic clinical characteristics of laboratory test - GraphROC for Windows. Scand. J. Clin. Lab. Invest. 55(222):1995;43-60.
McClish D.K. Analyzing a portion of the ROC curve. Med. Decis. Making. 9:1989;190-195.
Metz, C.E., Shen, J.-H, Herman, B.A., 1990. New methods for estimating a binormal ROC curve from continuously-distributed test results. Invited for presentation at the 1990 Joint Statistical Meetings of the American Statistical Society and the Biometric Society, Anaheim, CA.
Metz C.E., Herman B.A., Shen J.-H. Maximum likelihood estimation of receiver operating charcateristic (ROC) curves from continuously-distributed data. Med. Decis. Making. 17:1998;1033-1053.
National Mastitis Council, 1990. Microbiological Procedures for the Diagnosis of Bovine Udder Infection. Madison, WI.
Paape, M.J., Contreras, A., 1996. Historical perspective of the evolution of milk somatic cell count. Flem. Vet. J. 66 (Suppl.), 93-105.
Pepe M.S. Three approaches to regression analysis of receiver operating characteristic curve for continuous test results. Biometrics. 54:1998;124-135.
Pyörälä S., Pyörälä E. Accuracy of methods using somatic cell count and N-Acetyl-β-Glucosaminidase activity in milk to assess the bacteriological cure of bovine clinical mastitis. J. Dairy Sci. 80:1997;2820-2825.
SAS, 6.12. 1994. SAS User's Guide: Statistics. Cary, NC.
Serieys F. La numération des cellules du lait: Interprétation pour le diagnostic et le suivi des infections mammaires. Recueil de Médecine Vétérinaire. 161:1985;553-566.
Vecht U., Wisselink H.J., Defize P.R. Dutch national mastitis survey. The effect of herd and animal factors on SCC. Neth. Milk Dairy J. 43:1989;425-429.
Walter S.D., Irwig L.M. Estimation of test error rates, disease prevalence, and relative risk from misclassified data: a review. J. Clin. Epidem. 41(9):1988;923-937.
Zweig M.H., Campbell G. Receiver-opeating characteristic (ROC) plots: A fundamental evaluation tool in clinical medicine. Clin. Chem. 39(4):1993;561-577.
Zou K.H., Hall W.J., Shapiro D.E. Smooth non-parametric receiver operating characteristic (ROC) curves for continuous diagnostic tests. Stat. Med. 16:1997;2143-2156.
