AMA: Clinical authority of agentic AI should be “earned through evidence”

Key takeaways

The American Medical Association’s new one-page governance framework says agentic AI should earn clinical authority through evidence of safety and effectiveness, then keep it through ongoing performance, with accountability assigned in proportion to each tool’s risks. The framework poses four questions for developers and deployers: what clinical function the AI performs, what proves it can do so and what happens on failure, what it’s permitted to do and under what conditions, and who owns the risk. It also calls for distributed liability across developers, deployers, health systems, physicians and payers, consistent with the AMA’s August letter to the Washington Medical Commission warning that putting full responsibility on clinicians would discourage AI use.

Agentic AI tools working in the clinical environment should have to earn their authority to do so through evidence of their safety and effectiveness — and maintain that authority through continued demonstration of acceptable performance, says the American Medical Association (AMA) in a new governance brief.

In a one-page infographic, the AMA presents a framework for agentic AI tools that assumes clinical functions, focusing on accountability and oversight aligned with the risks and responsibilities of any given system.

The framework encourages agentic AI developers and deployers to build a “continuous flywheel” of performance and oversight by asking four main questions about agentic tools intended to take on roles in clinical care:

What clinical function is the AI performing? AI adopters should clearly define the task at hand, the patient population served by the system, the setting in which it will operate, and the intended use of the tool.

What demonstrates that it can perform that function, and what happens if it fails? Implementers should evaluate the performance, safety, benefits, and population fit “in proportion to potential consequence,” the AMA says, to ensure the tool is appropriate for the use case.

What should the AI be permitted to do, and under what conditions? Setting clear guardrails around the “permissible scope of action” is essential, as is defining who holds authority and oversight so that liability is clear if a negative event occurs.

Who is responsible for the risk and outcomes? Developers and deployers should assign accountability according to “authority, control, knowledge of risk, and ability to prevent or mitigate harm,” says the framework, emphasizing the need for layers of responsibility rather than assigning all accountability to the clinical user. 

The AMA intends to use these foundational questions to guide future approaches to responsible governance, especially around issues of liability, licensure, and regulation.

Interestingly, the framework includes a footnote on the use of the term “licensure,” which the AMA says does not mean licensure to practice medicine. Instead, the term refers to “authorization for an AI system to operate in a defined clinical capacity,” which is a concept that the broader industry has not yet fully addressed.

In August, the AMA sent a letter to the Washington Medical Commission supporting the Commission’s proposed policy statement that AI itself cannot hold a license to practice medicine, and that licensed clinicians must retain “the final authority on patient care.”

However, the same letter took issue with language in the Commission’s policy statement that places full responsibility on those same licensed human clinicians for errors or harms caused by an AI tool.

The new agentic AI framework specifically advocates for distributed accountability and liability, pointing out that developers, deployers, health systems, physicians, payers, and other actors should all be involved in accountability decisions.

In the August letter, AMA CEO John Whyte, MD, MPH, cautioned that placing full responsibility of potential harm onto the clinical end-user could limit adoption.  

“[A] physician who is told that they will bear full responsibility for a diagnostic miss produced by a flawed algorithm they did not know or have reason to know was experiencing a performance issue has a straightforward incentive: use AI tools as little as possible, or not at all,” wrote Whyte.

A more nuanced approach to oversight could help to avoid this situation, and the AMA framework adds to the conversation around what that approach should look like.

Naturally, authority and oversight will evolve as agentic AI technology continues to mature, the Association says, but these principles should always remain aligned in service of better patient care.Jennifer Bresnick is a journalist and freelance content creator with a decade of experience in the health IT industry.  Her work has focused on leveraging innovative technology tools to create value, improve health equity, and achieve the promises of the learning health system.  She can be reached at [email protected].