In 2021, I-MED Radiology Network successfully implemented Annalise Enterprise CXR, an AI-tool intended to assist clinicians with the interpretation of chest X-ray images, across most of its 250+ sites. With release of the Annalise Enterprise CTB module for non-contrast head CT studies, I-TeleRAD, part of the I-MED Radiology Network, was the first health care institution to implement and use the tool in their clinical routine. Their experience with the Annalise Enterprise CTB module exceeded prior expectations, followed by a roll-out across the I-MED Radiology Network.
Annalise Enterprise CT Brain Module
Can detect up to 130 suspected findings relevant to inpatient and outpatient settings.
Automatically corrects for rotation and tilt and reformats images in three planes in the Annalise Viewer, localising the finding and providing a measure of confidence.
Can offer worklist triage, helping to prioritise cases according to criticality.
Has a configurable user interface (UI) and integrates seamlessly into RIS/PACS.
Trained on one of the world’s largest radiologist hand- labelled, non-contract CTB (NCCTB) data sets of more than 229,000 studies.
Annalise.ai’s clinician-led approach
AI-tools can only reach their potential when fully adopted by clinicians in clinical practice. With a clinician-led approach, Annalise.ai aims to bring maximum value to the point of care. This means designing clinically relevant, easy-to-use, seamlessly integrated solutions with the clinician at the core, including:
- In-house clinical expertise and leadership within Annalise.ai.
- Close collaboration with customers and partners during product development.
- Close monitoring of product implementation at key customers, obtaining early feedback from clinical use.
- Early product optimisation from customer feedback, if needed, prior to a full commercial release.
First clinical implementation of Annalise Enterprise CT Brain at I-TeleRAD
The Annalise Enterprise CTB module was launched with a limited number of radiologists from I-TeleRAD, part of the I-MED Radiology Network. Between May and August 2022, 11 experienced radiologists (all >6 years of radiologist work experience) used the AI-tool for six weeks in routine clinical practice. They were closely trained and monitored, providing feedback on several standardised occasions. This allowed testing the performance of Annalise Enterprise CTB in normal use, gathering early feedback on features, performance of AI findings and product usability (Figure 1). More than 1,200 NCCTB studies were reported. The study design included three structured online surveys, user interviews and specific feedback about the performance of the AI findings integrated directly in the Annalise Viewer. Radiologists were able to reject suggested or add additional findings. This enabled further refinement of the operating thresholds for some findings, enhancing user satisfaction.
I-TeleRAD
- Part of I-MED Radiology Network
- 150 employed radiologists
- Access to a network of 350+ radiologists
- 24/7 availability
- Average reporting time for urgent cases: 33 minutes
Source: i-telerad.com.au
Prevailing positive attitude and high expectations regarding AI in radiology
Prior to the start of the study period, participants displayed positive expectations on the impact of AI in radiology, expecting AI assistance to improve patient outcomes and reporting accuracy. Positive expectations and attitude towards AI are common among radiologists and healthcare professionals and increase with AI experience.
believed that AI will have a
positive impact on radiology.
believed that AI assistance in
radiological practice will lead
to better patient outcome.
preferred reporting with AI
assistance as compared to
unassisted reporting.
Pre-Study Survey completed by 11 out of 11 participants
Easy to learn, simple to use
Participating radiologists quickly familiarised with the Annalise Enterprise CT Brain module, most reporting it took them 10 or fewer studies to feel comfortable with the tool, as reported in the Usability Survey (89% of respondents). All but one participant had previous experience with the Annalise Enterprise CXR module. Users specifically reported ease of use, potentially highlighting the benefit of comparable user interfaces across different modules of an enterprise solution.
“I haven’t had any issues… I’m also impressed at how good it is. You’ve taken not only the job of the software into consideration, which is to find abnormalities, but you’ve also taken the user interface into account. Which is just as important, because if the software is not easy to use, no one is going to use it.”
Dr Greg Compton
I-TeleRAD
Improving accuracy, decreasing reporting time
AI tools in clinical practice need to demonstrate added value to the clinical routine. Potential areas for benefit include improved clinical outcomes, accelerated processes and additional monetisation. While this study was not designed to quantify those effects, qualitative user feedback can be considered a strong indication and a prerequisite for full AI adoption in clinical routine.
After just 6 weeks of use
said that Annalise Enterprise
CTB improved reporting
accuracy, with no participants
reporting a decrease in accuracy.
said they were faster using
Annalise Enterprise CT Brain
(30% perceived no change in
reporting time).
would be disappointed if
Annalise Enterprise CT Brain
was taken away from their
practice.
Post-Study Survey completed by 10 out of 11 participants.
“It’s really helpful when you get tired. I use it to double check if I’ve missed something and it highlights the findings on the image very nicely for me. It’s a great reassurance and keeps you on your toes. It’s like a chaperone that improves my reporting, I absolutely love it.“
Prof Dr Jens Froelich
I-TeleRAD
Conclusion
In this first clinical product implementation, the Annalise Enterprise CT Brain module has proven to be quick to learn and an intuitive, comprehensive support tool for reporting non-contrast CT Brain findings. Radiologists experienced higher accuracy using Annalise Enterprise CTB, without compromising reporting speed. In contrast, all but one radiologist reported either no change or a decrease in reporting time, i.e., improved reading speed in addition to the accuracy gains. These results exceeded radiologists’ prior expectations on the benefits of the Annalise Enterprise CT Brain module (Figure 2).
Figure 2. Participants’ expectations regarding reporting speed (left) and accuracy (right) with AI assistance as compared to unaided reporting before implementation of Annalise Enterprise CTB (Pre-Study Survey, light purple, 11 respondents) and participants’ experiences after implementation and use of Annalise Enterprise CTB (Post-Study Survey, dark purple, 10 respondents).
As of November 2022, Annalise Enterprise CT Brain module is
available within most I-MED Radiology clinics.
- Scott I.A., Carter S.M., Coiera E. Exploring stakeholder attitudes towards AI in clinical practice. BMJ Health & Care Informatics 2021;28: e100450. doi: 10.1136/ bmjhci-2021-100450
- Van Leeuwen, K.G., de Rooij, M., Schalekamp, S. et al. How does artificial intelligence in radiology improve efficiency and health outcomes? Pediatric Radiology 52, 2087–2093 (2022). https://doi.org/10.1007/s00247-021-05114-8
The data presented is a result of a first clinical product implementation, including surveys, interviews and user feedback, as illustrated in Figure 1.This information is intended for health care professionals only. Annalise Enterprise is not intended to provide direct diagnosis. For detailed device information, including indications for use, contraindications, and warnings, please consult the user guide prior to use. Not all features are available in all regions. Check regulatory status with an Annalise.ai employee or contact info@annalise.ai. Annalise Enterprise CTB is not for sale in the U.S. Annalise.ai, ANNALISE ENTERPRISE and the Annalise logo are trademarks of Annalise-AI Pty Ltd, registered in Australia and other countries and regions.
Annalise-AI Pty Ltd., Level P, 24 Campbell Street, Sydney NSW 2000, Australia, Annalise.ai
OPT-PRM-065 Ver 0