After AI agents went rogue in lab tests, experts urge tighter controls on healthcare AI

Reports of AI agents acting outside human instruction are piling up. Since July, OpenAI, Anthropic and Google have each disclosed or confirmed that their models gained unauthorized access to outside systems.

OpenAI agents escaped a test environment in July and hacked into Hugging Face, an AI and machine learning platform. In September, the company said its agents had pulled public data from Securities and Exchange Commission and Census Bureau sites in ways it did not intend. Researchers at Transluce reported a failed attempt, apparently by OpenAI agents, to hack an Education Department website. Australia’s prime minister said an OpenAI agent broke into a government portal that holds Medicare statistics. The Australian government said the agent did not reach personal information.

Anthropic said in July that its models hacked into three organizations during testing. Google confirmed in September that its Gemini model did the same to three companies.

All of these incidents happened during the labs’ own testing and training, not in products sold to hospitals. But the technology is moving into administrative and clinical workflows across healthcare, and the incidents have sharpened the safety debate. In a September essay, Anthropic CEO Dario Amodei urged the industry to slow down so safety work can catch up.

DHI spoke to two physicians and a cybersecurity leader about what rogue AI behavior could mean for healthcare as adoption continues.

Oversight lags behind AI adoption in healthcare

AI is an appealing fix for long-standing problems such as clinician burnout and workforce shortages. “We’re primed for something to help us,” Peter Kowey, MD, a cardiologist and professor of medicine and clinical pharmacology at Thomas Jefferson University, told DHI.

Providers are building their own AI systems and tools. Vendors are adding AI to existing products and launching new ones. The pace leaves healthcare organizations struggling to govern the technology.

“Unfortunately, what’s happening in medicine is that it’s being implemented all over the place and in a very uncontrolled fashion with no regulations and no opportunity to find out whether it’s actually positively impacting what happens to patients,” Kowey said.

HIPAA covers the patient data that providers, health plans and clearinghouses feed to AI models. It does not say how healthcare organizations can use AI or who is accountable for the results.

Rogue behavior raises patient safety concerns

Replacing clinician judgment with an AI agent or another automated process is a hot-button issue in healthcare.

On September 30, six physician groups issued a joint statement on the role of AI in healthcare. It answered claims that AI is better informed than physicians. The American Academy of Family Physicians, American Academy of Pediatrics, American College of Obstetricians and Gynecologists, American College of Physicians, American College of Surgeons and American Medical Association signed it.

“Pitting physicians against technology or suggesting AI should replace physician expertise does not advance this important work. We need to move carefully, and we need to get this right,” the statement said.

Getting it right will require effective governance and a plan to respond if an AI system misbehaves, before it harms patients. Research already points to overreliance. In a July study in PLOS Digital Health, doctors trusted an AI’s incorrect patient classifications over the evidence in front of them, DHI reported in August.

“The scary part is that these systems will evolve over time in ways that we can’t even begin to predict,” Kowey said.

If that unpredictability includes the behavior seen in the lab incidents, where agents acted outside human instruction, patients could be harmed.

“They might, for example, when they’re given some directive from a clinician decide that’s probably not the best way to do this and change a therapeutic tactic or change the dose of a drug or change the drug that’s used as an antibiotic in a specific clinical situation without ever letting the practitioner know that they’re doing it,” Kowey said.

Liability and security could reshape AI adoption

Kowey expects litigation to be a turning point for how the industry approaches AI.

“Litigation is going to be the roadblock for AI,” Kowey said. “The liability is not going to be for the AI manufacturer. The liability is going to be for the purchaser of the system or the user of the system.”

Navin Sethi, MD, argues for a more selective approach to adoption. Sethi is a spinal surgeon and the chairman of surgery at Holy Cross Hospital in Silver Spring, Md. He is also chief medical officer of Zyter|TruCare, a healthcare software company.

“It’s selectively using AI in healthcare and deliberately thinking out a game plan,” he said. “This is what we’re going to use it for, not just saying, ‘Okay, we’re going to use it for everything.’”

Whether or not lawsuits arrive, healthcare organizations have to manage AI risk now. That means cutting through a lot of noise.

“I don’t think AI is where marketing is telling you it is right now. But I do believe that in time it will get there. And when it does, it’s better to be prepared,” said Joe Oleksak, a partner in the cybersecurity practice at Plante Moran, an audit, tax and consulting firm.

Being prepared means getting governance and security right. A healthcare organization needs controls that let it do five things:

• Ensure staff use AI tools safely
• Validate the tools’ outputs
• Respond if a tool goes rogue
• Demonstrate compliance with existing regulations
• Mitigate the cybersecurity risks that come with AI

“Business leaders who are taking AI seriously, therefore taking security seriously, will understand that throwing AI on top of the current model of security in healthcare probably isn’t a great idea,” Oleksak said. “More discipline is needed, more granularity is needed, more segmentation is needed and more oversight is needed.”

Carrie Pallardy, a Chicago-based freelance writer and editor, began her career covering healthcare more than a decade ago. Her work has taken into many different industries, but covering healthcare delivery remains a constant focus. She can be reached at [email protected] or on LinkedIn.