What does the tug of war between AI innovation and regulation mean for physicians and patients?

The American Medical Association first polled physicians on their health AI usage back in 2023. Three years later, usage has nearly doubled; the AMA reports that 81% of physicians are using AI in their practices. While adoption has increased, regulation and oversight trail behind.  

“We’re at a really interesting digital health revolution in which the technologies are not necessarily mature enough, the infrastructure is not necessarily mature enough, the regulation is not mature enough, yet we are being pushed to use it as much as possible and implement it as much as possible in our clinical practice,” David Toro-Tobon, MD, an investigator at the Care and AI Laboratory within the division of endocrinology at Mayo Clinic, told DHI.  

Early moves to regulate AI in healthcare exist across various federal agencies and in multiple forms at the state level. Policymakers are trying to walk a tightrope of regulating a quickly moving technology without hindering its innovation. These efforts have yet to produce any clear, unifying safety and accountability expectations for healthcare providers using AI tools. What does this mean for the physicians using AI tools and for the patients receiving care shaped by AI?  

Regulatory oversight  

At the federal level, regulatory oversight of AI has been mostly hands-off. President Trump has signed a series of executive orders on AI centered on innovation and American AI leadership. His latest executive order — Promoting Advanced Artificial Intelligence Innovation and Security — takes a step toward more oversight when it comes to AI and cybersecurity. The order mandates the creation of a voluntary framework that would allow the government to review new AI models before their developers launch them.  

Yet, there is also a deregulatory push at the federal level. Under Department of Health and Human Services (HHS) Secretary Robert F. Kennedy, Jr., proposed rules from the Office of the National Coordinator for Health Information Technology (ONC) would weaken AI transparency requirements, KFF Health News reports.  

The HTI-1 Final Rule, which went into effect in 2024, requires “model cards” for AI tools certified by the ONC.  

“People often compared it to a nutrition label for clinical AI,” said Maya Sandalow, associate director for the health program at the Bipartisan Policy Center. “This latest suite of ONC proposed rules would eliminate that model card or nutrition label requirement entirely. And that’s part of a broader deregulatory package.” 

The model card requirement was designed to help healthcare providers evaluate AI tools before buying them. Sandalow noted that the requirement was in effect for just a short time and was not actively enforced.  

The Food and Drug Administration (FDA) is another key regulator, but its role in overseeing AI use in healthcare is hardly finalized, according to Sandalow.  

“So far, FDA has only reviewed fixed AI enabled medical devices, but of course the technology exists for what are called adaptive tools that can evolve based on updated data and their training environment,” she said. “Generative AI has not yet been cleared or reviewed by the FDA, but they’re actively thinking about it.”   

The Centers for Medicare and Medicaid Services (CMS) is another significant federal player when it comes to how AI is deployed in healthcare.  

“If our goal is to incentivize adoption of safe and effective health AI tools and get them in the hands of doctors and patients who need them, we don’t just need regulatory guardrails, we also need incentive and the ability for these tools that are actually really high value to get to market,” said Sandalow. “Payment is a big piece of that puzzle.” 

While AI oversight efforts are slowly being shaped at the federal administrative and legislative levels — the White House released a National Policy Framework for Artificial Intelligence in March — individual states are taking varied approaches to regulating AI.  

The Trump administration argues that state regulation “creates a patchwork of 50 different regulatory regimes that makes compliance more challenging, particularly for start-ups.

Some kind of federal standard has bipartisan support, but Sandalow points out that agreeing what that standard should look like is easier said than done. “The devil is in the details, and the ability to form agreement…gets harder and harder the more detailed you get.”  

The gaps in AI oversight  

It takes time to hammer out policy. Legislators need time to understand AI. Regulatory agencies have ways to oversee AI, but as Toro-Tobon pointed out, “None of these pathways were originally designed for artificial intelligence.” 

In the meantime, AI adoption is hardly going to slow down. For now, that leaves healthcare providers with no industry-wide standard for evaluating AI tools.  

“The concern is leaving these practices to independently assess these AI tools and also to assume the risk and liability associated with them,” said Sandalow.  

And then there is the question of continuous monitoring. Are healthcare organizations prepared to audit the safety and efficacy of AI tools in clinical use over time?  

“We know that models can drift; outcomes can change. And so even with tools that have been previously validated, you need to continue to monitor to ensure…performance,” Edward Lee, MD, MPH, a practicing internal medicine physician and chief medical officer of Nabla, a company that provides ambient AI tools for clinicians, said. 

Clinician in the loop is frequently cited as an AI oversight solution, one that Toro-Tobon and his colleague argued is flawed in a recent piece published in The BMJ. This solution positions physicians as the experts responsible for the outcomes related to the use of an AI tool.  

“Most physicians don’t really receive any training on how to dissect AI outputs and how to understand or translate the AI output,” he said. “Even though we’re asking physicians to be that safety valve, our workforce is not really ready to assume that responsibility.” 

Toro Tobon argues that accountability should not rest solely with the physician, rather it should be shared with the companies that develop the tools and the health systems that put them in the hands of clinicians.  

The outlook for physicians and patients 

The enthusiasm for AI adoption in healthcare is fueled by its powerful potential to reduce physicians’ administrative burden and improve patient care. Health systems are building these tools themselves and adopting solutions from vendors crowding the market. But as adoption without clear safety, efficacy, accountability and monitoring standards continues, physicians and patients shoulder the risks.   

Physicians could find themselves accountable for outcomes influenced by tools they do not fully understand.  

“The risks are that we become very reliant on tools that may not necessarily be the best for patient care, that we don’t have, necessarily, evidence to show just yet that there are high quality outputs that are being produced by the tools,” Lee said.  

That risk gets passed onto patients who have little choice, or even knowledge, if AI tools are being used by their healthcare providers, ostensibly to improve their care.   

“If we have systems that are not safe or not fit for purpose, then we are achieving the complete opposite,” Toro-Tobon said. “Instead of helping us make a good decision, it can lead to potential clinical errors.” 

Carrie Pallardy, a Chicago-based freelance writer and editor, began her career covering healthcare more than a decade ago. Her work has taken into many different industries, but covering healthcare delivery remains a constant focus. She can be reached at [email protected] or on LinkedIn.