ArXiv Preprint
Objective: Reproducibility is critical for translating machine learning-based
(ML) solutions in computational pathology (CompPath) into practice. However, an
increasing number of studies report difficulties in reproducing ML results. The
NCI Imaging Data Commons (IDC) is a public repository of >120 cancer image
collections, including >38,000 whole-slide images (WSIs), that is designed to
be used with cloud-based ML services. Here, we explore the potential of the IDC
to facilitate reproducibility of CompPath research.
Materials and Methods: The IDC realizes the FAIR principles: All images are
encoded according to the DICOM standard, persistently identified, discoverable
via rich metadata, and accessible via open tools. Taking advantage of this, we
implemented two experiments in which a representative ML-based method for
classifying lung tumor tissue was trained and/or evaluated on different
datasets from the IDC. To assess reproducibility, the experiments were run
multiple times with independent but identically configured sessions of common
ML services.
Results: The AUC values of different runs of the same experiment were
generally consistent and in the same order of magnitude as a similar,
previously published study. However, there were occasional small variations in
AUC values of up to 0.044, indicating a practical limit to reproducibility.
Discussion and conclusion: By realizing the FAIR principles, the IDC enables
other researchers to reuse exactly the same datasets. Cloud-based ML services
enable others to run CompPath experiments in an identically configured
computing environment without having to own high-performance hardware. The
combination of both makes it possible to approach the reproducibility limit.
Daniela P. Schacherer, Markus D. Herrmann, David A. Clunie, Henning Höfener, William Clifford, William J. R. Longabaugh, Steve Pieper, Ron Kikinis, Andrey Fedorov, André Homeyer
2023-03-16
