Mass General Brigham is a leading provider of integrated, accessible, and equitable patient-centered healthcare across the United States. As part of their commitment to delivering top-quality care, they collaborate in research to bring new medical discoveries to the bedside – for the benefit of the communities they serve and future generations. In 2021, Mass General Brigham Data Science Office investigated an innovative artificial intelligence (AI) decision-support solution for chest X-rays using data from four sites across the Mass General Brigham network. The study aimed to compare the accuracy of Annalise Enterprise CXR Triage Pneumothorax to thoracic radiologist interpretations in detecting pneumothorax and tension pneumothorax cases.
Mass General Brigham
Annalise Enterprise CXR Triage Pneumothorax
- Is one of the only FDA-cleared AI-tools that differentiates pneumothorax and tension pneumothorax in chest X-rays*
- Is one of the only decision-support AI-tools for chest X-rays that analyzes up to three images, including laterals*
- Features a customizable user interface (UI) that can integrate seamlessly into RIS/PACS
- Helps triage pneumothorax and tension pneumothorax cases through worklist prioritization
- Was developed on one of the world’s largest chest X-ray datasets of more than 780,000 studies hand-labeled by consultant radiologists
Study background
The early detection of pneumothorax – commonly on chest radiograph (CXR) – helps to guide clinical decision making. In cases of tension pneumothorax, lack of timely and effective treatment can lead to patient deterioration and death. The ability to accurately detect and rapidly triage pneumothorax with an AI model could improve patient outcomes by aiding earlier reporting, diagnosis, and intervention by clinicians. Annalise Enterprise CXR Triage Pneumothorax is one of the only FDA-cleared solutions that detects and differentiates both simple and tension pneumothorax*.
Objective
This study aimed to compare the accuracy of an AI model (Annalise Enterprise CXR Triage Pneumothorax) to thoracic radiologist interpretations in detecting pneumothorax and tension pneumothorax.
Methods
A retrospective assessment was conducted on a dataset of 1,000 CXR cases obtained from four hospitals in Mass General Brigham’s network. All cases involved inpatients or outpatients aged 18 years or older. They were selected using two strategies:
- Identifying consecutive pneumothorax cases through a manual review of radiology reports.
- Identifying consecutive tension pneumothorax cases using natural language processing.
Negative cases were selected by taking the next negative case from the same X-ray machine as each positive case. Each case was independently assessed by up to three radiologists to establish consensus interpretations. Each case was then interpreted by the AI model for the presence of pneumothorax or tension pneumothorax.
Outcome measures
The primary endpoint was the area under the receiver operating characteristic curves (AUCs) for the detection of pneumothorax and tension pneumothorax. Secondary endpoints were the sensitivities and specificities for the detection of pneumothorax and tension pneumothorax at predefined operating points.
Key figures
4 sites
1,000 CXRs analyzed
- 3 cases excluded due to poor image quality
- 12 cases excluded due to unsuccessful model inference
985 CXRs in the final dataset
- 307 non-tension pneumothorax
- 128 tension pneumothorax
- 550 negative cases
Results
The AI model detected pneumothorax:
AUC of AUC of 0.979
Sensitivity 94.3%
Specificity 92.0%
The AI model detected tension pneumothorax:
AUC of AUC of 0.987
Sensitivity 94.5%
Specificity 95.3%
Importance of findings
Annalise Enterprise CXR Triage Pneumothorax accurately detected pneumothorax and tension pneumothorax on this dataset of 1,000 CXRs. The solution surpassed recognized accuracy benchmarks (>0.95 AUC, >80% sensitivity and specificity). These results obtained on a US external dataset are consistent with previously published performance of the model, suggesting that the model is generalizable across different populations and clinical settings. Annalise Enterprise CXR Triage Pneumothorax is like a second pair of eyes, supporting the identification of pneumothorax, empowering clinicians to make fast, accurate decisions. Its use in clinical workflows could lead to earlier identification and improved care for patients with pneumothorax.
- Zarogoulidis P, Kioumis I, Pitsiou G, et al. Pneumothorax: from definition to diagnosis and treatment. J Thorac Dis. 2014;6(Suppl 4):S372-S376. doi:10.3978/j.issn.2072-1439.2014.09.24
Information contained herein is for distribution in the US only. The device is intended for use by healthcare professionals only. The device is not intended to provide direct diagnosis. For detailed device information, including indications for use, contraindications, and warnings, please consult the user guide prior to use.
Annalise-AI Pty Ltd., Level P, 24 Campbell Street, Sydney NSW 2000, Australia
OPT-PRM-064 V1