Artificial Intelligence-Enabled Medical Devices: The Importance of Clinical Validation
Artificial intelligence (AI) has revolutionized the field of medical devices, offering innovative solutions for diagnosis and treatment. However, a recent study published in JAMA Health Forum has shed light on the importance of clinical validation for AI-enabled medical devices to ensure their safety and reliability.
The study, conducted by researchers from the Johns Hopkins Carey Business School, analyzed 950 AI medical devices authorized by the Food and Drug Administration (FDA) through November 2024. Shockingly, the study found that devices with no clinical validation were more likely to be the subject of recalls. Sixty of the devices analyzed were associated with 182 recall events, with the most common causes being diagnostic or measurement errors, followed by functionality delays or loss.
Lead author Tinglong Dai, a professor at the Johns Hopkins Carey Business School, emphasized that the majority of recalled devices had not undergone clinical trials. This is particularly concerning for devices that went through the FDA’s 510(k) pathway, as clinical studies are not required for these devices.
The study also revealed that devices that had undergone retrospective or prospective validation were subject to fewer recalls. Furthermore, publicly traded companies were found to account for disproportionately more recall events, with public company status associated with a nearly six times higher chance of a recall event. This discrepancy in recall rates between public and private companies highlights the need for stricter regulations and oversight in the AI medical device industry.
To address these concerns, the researchers recommended requiring human testing or clinical trials before a device is authorized. They also suggested incentivizing companies to conduct ongoing studies and collect real-world performance data to ensure the safety and effectiveness of AI-enabled medical devices.
In 2023, the FDA issued draft guidances to improve the 510(k) program, including recommendations on choosing appropriate predicate devices and when clinical data may be necessary to demonstrate substantial equivalence. However, these guidance documents are still pending finalization.
Overall, the findings of this study underscore the importance of clinical validation for AI-enabled medical devices to ensure their post-market safety and reliability. By implementing stricter regulations and incentivizing companies to conduct comprehensive studies, we can mitigate the risks associated with unvalidated AI medical devices and ultimately improve patient outcomes.
This study was a collaborative effort between researchers at the Johns Hopkins Carey Business School, the Johns Hopkins Bloomberg School of Public Health, and Yale School of Medicine, and was funded by an award from Johns Hopkins University.
