A groundbreaking new study published in npj Digital Medicine has exposed significant shortcomings in how the FDA tracks the safety of AI and machine learning medical devices after they reach the market. The research, conducted by experts from the University of Hong Kong, Harvard Law School, and the University of Potsdam, represents the first systematic assessment of the FDA’s postmarket surveillance system for AI/ML medical devices.

NyquistAI’s Role in Critical Research

The researchers relied on NyquistAI’s database to identify and analyze software-enabled medical devices, demonstrating the platform’s value for regulatory research and policy development. NyquistAI helped create the comprehensive dataset that made this analysis possible.

This collaboration showcases how NyquistAI’s platform enables researchers to:

• Identify patterns across large datasets of medical devices
• Support evidence-based policy recommendations
• Bridge the gap between regulatory data and research insights

What the Research Found

Using data from NyquistAI’s comprehensive database alongside FDA records, the researchers analyzed adverse event reports for approximately 950 AI/ML medical devices approved between 2010 and 2023. Their findings reveal troubling gaps in our current safety monitoring system:

Extreme Concentration of Problems: More than 98% of adverse events were concentrated in fewer than five devices—a much higher concentration than seen with traditional medical devices.

Massive Data Gaps: Critical information was missing across the board:

• Event location: 100% missing
• Health professional reporter status: 73% missing
• Event date: 32% missing
• Reporter occupation: 30% missing

Misclassified Events: Many reports labeled as device “malfunctions” were actually due to user error or analyst mistakes, not true AI/ML system failures.

The Unique Challenge of AI/ML Devices

The study highlights why traditional adverse event reporting falls short for AI/ML medical devices. Unlike conventional medical equipment, AI/ML systems can experience problems like:

• Concept drift: When the relationship between patient characteristics and outcomes changes over time
• Covariate shift: When the patient population using the device differs from the training data
• Algorithmic instability: When similar patients receive vastly different diagnoses or treatments

These AI-specific issues often manifest at the population level rather than in individual patient cases, making them nearly impossible to capture through traditional adverse event reporting.

What Needs to Change

The study proposes two paths forward:

1. Enhanced Reporting: Requiring manufacturers to submit quarterly reports on training data updates, deployment condition changes, and algorithmic stability issues
2. New Regulatory Approaches: Moving beyond traditional adverse event reporting to include “nutrition label” style disclosures for AI/ML devices, similar to what the Office of the National Coordinator recently finalized for predictive decision support tools

The Broader Impact

This research arrives at a crucial time, as the FDA’s newly established Digital Health Advisory Committee has made AI governance a top priority. With AI/ML medical devices proliferating rapidly, ensuring robust postmarket surveillance is essential for patient safety and public trust in these technologies.

The study’s authors have made their complete dataset and code publicly available, enabling further research and transparency in this critical area of medical device safety.