Comparative Empirical Evaluation of Monte Carlo Dropout Uncertainty Measures for Failure Capture and Referral Trade-offs in Chest X-ray Disease Detection: A Reproducible Simulation Study

Yali Zhang; Chenhui Hao

Comparative Empirical Evaluation of Monte Carlo Dropout Uncertainty Measures for Failure Capture and Referral Trade-offs in Chest X-ray Disease Detection: A Reproducible Simulation Study

Authors

Yali Zhang Computer Science, Rice University, Houston, TX, USA
Chenhui Hao Civil Engineering, University of California, CA, USA

Keywords:

uncertainty estimation, Monte Carlo dropout, chest radiography, selective prediction

Abstract

Reliable deployment of automated chest radiograph interpretation depends not only on classification accuracy but on whether a network can flag the cases it is likely to read incorrectly. This study presents a comparative evaluation of four uncertainty measures derived from Monte Carlo dropout, namely predictive confidence, predictive variance, predictive entropy, and mutual information, applied to two clinically motivated tasks: capturing misclassified predictions and supporting uncertainty-based referral. Because the real image collections were not available for this work, the comparison is conducted on a reproducible synthetic benchmark that emulates the statistical behaviour of a DenseNet-121 classifier under Monte Carlo dropout on five findings from a ChestX-ray14-style multi-label detection task, together with a smaller synthetic set that emulates a CheXpert-style radiologist-consensus partition; every reported number is computed from this simulated data rather than from real radiographs or real model inference. Misclassification detection is scored through the area under the receiver operating characteristic curve, while referral is analysed by withholding the most uncertain cases and measuring the sensitivity and specificity retained on the remainder. Predictive variance attains the highest misclassification-detection area under the receiver operating characteristic curve at 0.800, ahead of predictive entropy at 0.784, mutual information at 0.782, and predictive confidence at 0.743; on the precision–recall view the order shifts and predictive entropy leads at 0.612. Referring the most uncertain 20% of cases raises retained sensitivity from 0.770 to 0.783 and specificity from 0.704 to 0.734. The synthetic external set reproduces the same AUROC ordering. Within the limits of a simulation study, the evidence indicates that distribution-based measures offer a consistent advantage over single-pass confidence while isolating the evaluation protocol on data with a fully known ground truth; validation on real radiographs remains necessary before any clinical claim.

References

Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (PMLR 70, pp. 1321–1330).

Leibig, C., Allken, V., Ayhan, M. S., Berens, P., & Wahl, S. (2017). Leveraging uncertainty information from deep neural networks for disease detection. Scientific Reports, 7, 17816.

Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017). ChestX-ray8: Hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2097–2106).

Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C., Shpanskaya, K., Lungren, M. P., & Ng, A. Y. (2017). CheXNet: Radiologist-level pneumonia detection on chest X-rays with deep learning. arXiv. https://arxiv.org/abs/1711.05225

Nair, T., Precup, D., Arnold, D. L., & Arbel, T. (2020). Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation. Medical Image Analysis, 59, 101557.

Geifman, Y., & El-Yaniv, R. (2017). Selective classification for deep neural networks. In Advances in Neural Information Processing Systems (Vol. 30, pp. 4878–4887).

Gal, Y., & Ghahramani, Z. (2016). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning (PMLR 48, pp. 1050–1059).

Gal, Y., Islam, R., & Ghahramani, Z. (2017). Deep Bayesian active learning with image data. In Proceedings of the 34th International Conference on Machine Learning (PMLR 70, pp. 1183–1192).

Kendall, A., & Gal, Y. (2017). What uncertainties do we need in Bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems (Vol. 30, pp. 5574–5584).

Depeweg, S., Hernández-Lobato, J. M., Doshi-Velez, F., & Udluft, S. (2018). Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning. In Proceedings of the 35th International Conference on Machine Learning (PMLR 80, pp. 1184–1193).

Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems (Vol. 30, pp. 6402–6413).

Hendrycks, D., & Gimpel, K. (2017). A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations.

Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J. V., Lakshminarayanan, B., & Snoek, J. (2019). Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems (Vol. 32, pp. 13969–13980).

Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., Seekins, J., Mong, D. A., Halabi, S. S., Sandberg, J. K., Jones, R., Larson, D. B., Langlotz, C. P., Patel, B. N., Lungren, M. P., & Ng, A. Y. (2019). CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 33, pp. 590–597).

Huang, G., Liu, Z., van der Maaten, L., & Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2261–2269).

Band, N., Rudner, T. G. J., Feng, Q., Filos, A., Nado, Z., Dusenberry, M. W., Jerfel, G., Tran, D., & Gal, Y. (2021). Benchmarking Bayesian deep learning on diabetic retinopathy detection tasks. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track.

Nguyen, H. Q., Lam, K., Le, L. T., Pham, H. H., Tran, D. Q., Nguyen, D. B., Le, D. D., Pham, C. M., Tong, H. T. T., Dinh, D. H., Do, C. D., Doan, L. T., Nguyen, C. N., Nguyen, B. T., Nguyen, Q. V., Hoang, A. D., Phan, H. N., Nguyen, A. T., Ho, P. H., & Vu, V. (2022). VinDr-CXR: An open dataset of chest X-rays with radiologist's annotations. Scientific Data, 9, 429.

Johnson, A. E. W., Pollard, T. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-y., Peng, Y., Lu, Z., Mark, R. G., Berkowitz, S. J., & Horng, S. (2019). MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs. arXiv. https://arxiv.org/abs/1901.07042

Bustos, A., Pertusa, A., Salinas, J.-M., & de la Iglesia-Vayá, M. (2020). PadChest: A large chest X-ray image dataset with multi-label annotated reports. Medical Image Analysis, 66, 101797.

Comparative Empirical Evaluation of Monte Carlo Dropout Uncertainty Measures for Failure Capture and Referral Trade-offs in Chest X-ray Disease Detection: A Reproducible Simulation Study