Independent retrospective evaluation of seven decision-support AI-tools on chest X-rays
The number of AI products purporting to assist with lung cancer detection has exploded in recent years1, however not all solutions provide the same level of transparency of their performance.2 With limited information to evaluate tools, decision makers often have to rely on informed reasoning when choosing AI solutions. Head-to-head comparisons constitute an important instrument to evaluate the evolving AI landscape.
From conjecture to certainty: AI as accurate as human
readers – or better
Facilitating fair and repeatable head-to-head comparison of AI products
A peer-reviewed head-to-head study, recently published in the prestigious journal Radiology, set out to evaluate the stand-alone performance of commercially available, decision-support AI-solutions in radiology. An independent, retrospective, multi-reader, multi-centre, multi-solution design was used.3 Lung nodule detection on chest radiographs was selected as one of two use cases.
Key facts of the Project AIR study3
- 386 unique CXR studies
- 14 vendors invited
- 7 vendors participated
- 17 human readers
- Controlled case pathology for lung nodules
- Ground-truthed by thoracic specialists with access to chest CT
About Project AIR
Project AIR is run by the Diagnostic Image Analysis Group, an independent research group at Radboud University Medical Center (Radboudumc), Netherlands. The group also maintains a comprehensive overview of available AI based software for clinical radiology practice, aiding healthcare professionals to easily find and compare the dynamic radiology AI market. Find out more at grand-challenge.org/aiforradiology/.
“Objective head-to-head comparison on a benchmark set can be helpful for choosing the right AI product and could even facilitate reimbursement decisions. That’s why we set up Project AIR.”
Dr Kicky van Leeuwen
Project Lead Project AIR | AI for Radiology | Radboudumc
Key findings of the Project AIR study3
Annalise Enterprise CXR – Adding onto existing evidence on
super-human performance
Vigorous pre-market validation of Annalise Enterprise CXR
Annalise Enterprise CXR, one of the solutions under investigation in the Project AIR study3, is a comprehensive decision-support AI solution, capable of detecting up to 124 findings on chest X-rays. Annalise solutions are designed by clinicians for clinicians, vigorously trained and extensively validated before their availability for clinical use. The outstanding performance in the Project AIR study did not come as a surprise, but supports existing evidence from a large multi-reader, multi-center (MRMC) validation study on Annalise Enterprise CXR.4
MRMC validation study setup4
- 2,568 unique CXR studies
- 20 fully-trained radiologists
- 3 month washout period between reads
- ground-truthed by 3 thoracic radiologists
- 124+ findings, incl. solitary lung nodule, solitary lung masses, and multiple masses or nodules
Outperforming human readers across
100+ findings
Annalise Enterprise CXR significantly outperformed human readers, both un-assisted and assisted by the AI model averaged across >120 findings (Figure 1). For single findings, AI performance was superior to human readers for 102 findings, including lung nodule and masses related findings (Figure 1). The model was non-inferior for an additional 19 findings.4 Notably, AI model standalone performance was comparably high across both Project AIR3 and the validation study4, underpinning the model’s robust performance across settings and cases.
Figure 1. Macro-average AUC for three lung cancer associated findings (solitary lung nodule, solitary lung mass, multiple nodules or masses; left) and all findings covered by Annalise Enterprise CXR (right) for unassisted radiologists (without AI), AI-assisted radiologists (with AI), and model alone.4
Boosting radiologists’ performance with
AI support
Importantly, medical imaging AI such as Annalise Enterprise CXR is usually not intended to work autonomously or provide direct diagnosis, but is designed as diagnostic-assist devices. Thus, AI-support is the most realistic clinical use case. Compared to unassisted radiologists, Annalise Enterprise CXR support resulted in:
+45%
higher accuracy averaged across all 124 findings (Figure 1)
+58%
higher accuracy for 3 findings indicative of lung cancer (Figure 1)
“We’re very excited about the new findings, but we weren’t all that surprised by them. From the outset, the team at Annalise.ai has been committed to building solutions backed by rigorous research. It’s great to prove we’re achieving what we’ve been striving for.”
Dr Mark Phillips
Head of Research & Medical Affairs | Annalise.ai
About Annalise Enterprise CXR
Capable of detecting up to 124 findings in seconds, Annalise Enterprise CXR is a comprehensive decision support solution for chest radiographs. Designed by clinicians for clinicians, it is like a second pair of eyes that never get tired, providing a reassuring safety net in busy clinical settings. Cases can be reported faster, saving time that can be spent elsewhere in the clinical workflow.
1 AI for Radiology, https://grand-challenge.org/aiforradiology/. Accessed 10/07/23.
2 Leeuwen KG van, Schalekamp S, Rutten MJCM, et al. Artificial intelligence in radiology: 100 commercially available products and their scientific evidence. Eur Radiol 2021; 31:3797–3804.
3 Leeuwen, KG van et al. Comparison of commercial AI software performance for radiograph lung nodule detection and bone age prediction. Radiology 2024; 310(1):e230981.
4 Seah JCY et al. Effect of a comprehensive deep-learning model on the accuracy of chest x-ray interpretation by radiologists: a retrospective, multireader multicase study. Lancet Digit Health 2021;
3(8):e496-e506.
This information is intended for health care professionals only. Annalise Enterprise is not intended to provide direct diagnosis. For detailed device information, including indications for use, contraindications, and warnings, please consult the user guide prior to use. Annalise Enterprise CXR is a class IIb device under Regulation (EU) 2017/745. Not all features are available in all regions. Check regulatory status with an Annalise.ai employee or contact info@annalise.ai. Annalise Enterprise CXR is not for sale in the U.S. Annalise.ai, ANNALISE ENTERPRISE and the Annalise logo are trademarks of Annalise-AI Pty Ltd, registered in Australia and other countries and regions.
Annalise-AI Pty Ltd., Level P, 24 Campbell Street, Sydney NSW 2000, Australia, Annalise.ai
OPT-PRM-102 Ver 1