Misuse of AI chatbots tops 2026’s health tech risk list

Excited about the new array of AI options to help patients and providers dig deeper into health records? Maybe hit the pause button on relying too heavily on these tools, which can sometimes put a very convincing veneer of authority around misinformation or suspect advice. 

Chatbots based on large language models (LLMs), such as ChatGPT, Claude, Amazon’s new One Medical AI for Health, and even Elon Musk’s sycophantic pet project Grok, can pose significant risks to patients if used improperly to guide decision making, warns ECRI’s Top Technology Hazards report for 2026. 

“LLM responses derive from predictions based on large datasets, not from genuine comprehension of the information,” the annual report says. “This can lead to ‘hallucinations’ (incorrect or misleading responses) or distortions from biases embedded in the dataset. These limitations, along with the models’ predisposition toward pleasing the user (more so than providing an accurate response), can cause a model to provide incorrect medical advice.” 

There are already a number of examples of LLMs doing just that, from mishandling sensitive mental health situations to inventing non-existing body parts to making major mistakes in the world of women’s health. 

ECRI adds that when staff members asked one chatbot about whether to place an electrosurgical return electrode over a patient’s shoulder blade, the tool incorrectly stated that the placement was appropriate. Had a clinician followed that advice, the patient would be at risk of burns.  

Now, as companies like OpenAI, Anthropic, and Amazon promote spaces with enhanced privacy and security that are dedicated to health queries (but based on the same models as have made egregious errors in the past), the likelihood of harm could increase – especially among communities that are being priced out of health coverage and might be turning to low-cost AI tools as an alternative source of advice.  

See where healthcare AI is going next — connect with the leaders turning innovation into impact at ViVE. AI Zone @ViVE 2026.

“AI models reflect the knowledge and beliefs on which they are trained, biases and all,” said Marcus Schabacker, MD, PhD, president and chief executive officer of ECRI. “If healthcare stakeholders are not careful, AI could further entrench the disparities that many have worked for decades to eliminate from health systems. Realizing AI’s promise while protecting people requires disciplined oversight, detailed guidelines, and a clear-eyed understanding of AI’s limitations.” 

With regulatory authorities continually stripping back even the minimal existing oversight of AI-powered health tools, including many medical devices and clinical decision support applications, healthcare providers might be faced with a new wave of misinformed patients – and the harmful clinical effects of bad advice from unregulated chatbots. 

With more than 230 million people already asking health-related questions to ChatGPT every week, according to OpenAI, it’s possible that the ship has already sailed on steering the public away from using these potentially untrustworthy sources as a primary avenue for health information. 

However, providers and other healthcare stakeholders can still work to offer education on safe and effective use of AI chatbots and create opportunities for dialogue with patients when something seems to be amiss with their thinking.  

Staying aware of the potential for misuse, even among clinicians who are availing themselves of enterprise AI chatbot tools, will be essential for identifying emerging misinformation and preventing poor outcomes from overreliance on technology. 

Other notable digital hazards on the 2026 list include unpreparedness for a “digital darkness” event in which providers experience a sudden loss of access to electronic systems and patient information; underutilizing medication safety technologies in the perioperative environment; cybersecurity risks from legacy medical devices; and health technology implementations that lead to unsafe clinical workflows. 

Jennifer Bresnick is a journalist and freelance content creator with a decade of experience in the health IT industry.  Her work has focused on leveraging innovative technology tools to create value, improve health equity, and achieve the promises of the learning health system.  She can be reached at [email protected].