Receive a weekly summary and discussion of the top papers of the week by leading researchers in the field.

Understanding metric-related pitfalls in image analysis validation.

In ArXiv

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

Reinke Annika, Tizabi Minu D, Baumgartner Michael, Eisenmann Matthias, Heckmann-Nötzel Doreen, Kavur A Emre, Rädsch Tim, Sudre Carole H, Acion Laura, Antonelli Michela, Arbel Tal, Bakas Spyridon, Benis Arriel, Blaschko Matthew, Büttner Florian, Cardoso M Jorge, Cheplygina Veronika, Chen Jianxu, Christodoulou Evangelia, Cimini Beth A, Collins Gary S, Farahani Keyvan, Ferrer Luciana, Galdran Adrian, van Ginneken Bram, Glocker Ben, Godau Patrick, Haase Robert, Hashimoto Daniel A, Hoffman Michael M, Huisman Merel, Isensee Fabian, Jannin Pierre, Kahn Charles E, Kainmueller Dagmar, Kainz Bernhard, Karargyris Alexandros, Karthikesalingam Alan, Kenngott Hannes, Kleesiek Jens, Kofler Florian, Kooi Thijs, Kopp-Schneider Annette, Kozubek Michal, Kreshuk Anna, Kurc Tahsin, Landman Bennett A, Litjens Geert, Madani Amin, Maier-Hein Klaus, Martel Anne L, Mattson Peter, Meijering Erik, Menze Bjoern, Moons Karel G M, Müller Henning, Nichyporuk Brennan, Nickel Felix, Petersen Jens, Rafelski Susanne M, Rajpoot Nasir, Reyes Mauricio, Riegler Michael A, Rieke Nicola, Saez-Rodriguez Julio, Sánchez Clara I, Shetty Shravya, van Smeden Maarten, Summers Ronald M, Taha Abdel A, Tiulpin Aleksei, Tsaftaris Sotirios A, Calster Ben Van, Varoquaux Gaël, Wiesenfarth Manuel, Yaniv Ziv R, Jäger Paul F, Maier-Hein Lena

2023-Feb-09