As generative AI moves toward performing increasingly consequential clinical functions, the American Medical Association (AMA) is urging the U.S. Food and Drug Administration to address a foundational question: Who is responsible for ensuring that these tools remain safe once they enter clinical practice?
In a letter released last week, AMA CEO and executive vice president Dr. John Whyte responded to the FDA’s request for feedback on the discussion paper published by the agency this August. The letter outlined the AMA recommendations for modernizing the FDA oversight of generative AI-enabled medical devices, emphasizing the need for robust clinical evidence, ongoing performance monitoring across use cases, and physician involvement in defining appropriate standards of care.
The FDA’s paper defined generative AI-enabled medical devices as “a class of AI models that emulate the structure and characteristics of input data in order to generate derived synthetic content. This can include images, videos, audio, text, and other digital content.” This broad category could extend to many technologies, including but not limited to software platforms, wearable tools, and clinical devices.
For Whyte, the debate goes beyond how AI-powered devices should be tested. He emphasized the need for clear accountability pathways when these devices fail, particularly in the cases where the device in question is patient-facing.
“If this tool fails or gives me wrong information, how does that impact my care?”, Whyte said. “We’re not addressing that enough.”
Both the FDA’s discussion paper and the response from the AMA reflect the emphasis on risk management. The FDA’s paper explored potential approaches to evaluating generative AI systems, including competency-based testing, clinical confirmation, and post-market monitoring.
The AMA’s perspective detailed two primary points, calling on the FDA to assure safety and effectiveness above regulatory agility in the face of rapid innovation, and asserting that physician expertise should inform the standards of regulation, rather than serve as the safety net.
The FDA’s discussion paper provided an example of where risk-based regulation would be particularly relevant.
“The risk of a software function also depends on the severity of harm that may result from using and relying upon an incorrect output,” the report stated. “An incorrect recommendation about an over-the-counter remedy for a minor symptom is meaningfully different from an incorrect recommendation about adjusting an insulin dose.”
Whyte also stated the care should be taken to offer the same standard of care across all settings. For example, if a patient is in a rural setting and only has access to an AI-enabled mode of medical care, they may have a different outcome than a patient in a better-resourced area who was able to interact with a human healthcare provider, particularly if the AI device in question did not undergo adequate regulation before deployment.
The AMA letter to the FDA, signed by Whyte, stated “We recognize the important opportunity for GenAI-enabled medical devices to expand access to clinical capabilities where appropriate care is unavailable, delayed, or difficult to obtain. The availability of alternative care is, therefore, a legitimate consideration in assessing the benefits and risks of a device. FDA should exercise significant caution, however, before allowing limitations in access to care to justify lower standards of device performance.”
Whyte went on to discuss the salient issue of liability.
“In some ways, I’ve argued these tools are ascribing more liability to physicians. Who’s responsible for risk management?,” Whyte said. “Right now, it’s the physician, and I’d argue we really have to think that through. We really do have to have a structured regulation around risk.”
The AMA letter concluded with an offer to collaborate with the FDA and other stakeholders “to help recognize the potential of AI to improve the health of our nation”. The challenge which lies ahead is not simply determining whether AI can perform a clinical task, but establishing a firm framework for who authorizes that task, who is held liable for the outcomes, what evidence supports those decisions, and who monitors AI-enabled medical devices’ performance over time and across an abundance of real-world settings in order to ensure innovation that truly supports the health of the nation.
Photo: Carol Yepes, Getty Images
