The Food and Drug Administration (FDA) is currently seeking public feedback on how to measure and evaluate the real-world performance of artificial intelligence-enabled medical devices. This consultation, which began on Tuesday, includes six sets of questions about ways to perform ongoing, systematic performance monitoring to assess how AI behaves in clinical settings.
As the rate of AI submissions continues to accelerate, the FDA is concerned about the potential for “data drift” to cause devices to perform worse in real-world scenarios than in the tests conducted to support their authorization. The agency’s call for feedback is based on evidence that the performance of AI-enabled medical devices can change over time due to factors such as changes in clinical practice, patient demographics, data inputs, and healthcare infrastructure. User behavior, workflow integration, and changes to clinical guidelines can also impact device performance.
Data drift, commonly known as the tendency for AI-enabled devices to change over time, can degrade performance, introduce bias, or reduce reliability. This phenomenon distinguishes AI-enabled devices from traditional products, whose performance should remain constant. In a draft guidance released in January, the FDA recommended that developers have a post-market monitoring plan in place for AI-enabled devices to address potential performance degradation over time.
Regulatory processes designed for traditional devices may establish a baseline for performance but may not predict how AI-enabled products will behave in dynamic, real-world environments. The FDA is seeking feedback on how to ensure the ongoing safety and effectiveness of AI-enabled devices throughout their lifecycles by adopting approaches to detect, assess, and mitigate performance changes over time.
While FDA officials have not yet proposed or implemented changes to the evaluation of AI-enabled devices, they are framing the consultation as an opportunity for external stakeholders to share insights and drive a broader discussion on the ongoing evaluation of AI in real-world settings. The agency has posed questions about performance metrics, real-world evaluation methods, post-market data, triggers for additional assessments, interactions between humans and AI, and best practices. The FDA is particularly interested in practical ways to measure and evaluate the performance of AI-enabled medical devices in clinical environments.
Through this consultation, the FDA aims to identify strategies for detecting and managing performance drift, such as monitoring changes in input and output. The agency is specifically looking for methods that are currently deployed at scale in clinical settings and supported by real-world evidence. The consultation period closes on December 1st.
Overall, the FDA’s initiative to gather feedback on evaluating the real-world performance of AI-enabled medical devices underscores the importance of ensuring the safety and effectiveness of these innovative technologies in clinical practice. By soliciting input from various stakeholders, the FDA is taking proactive steps to address the evolving challenges associated with AI in healthcare.
