ChatGPT Health, Amazon Health AI, and Claude for Healthcare are moving from health advice into the care pathway, and hospitals get little say

At a glance

A Nature Health paper argues that consumer AI from OpenAI, Amazon, Anthropic and Ant Group has become healthcare infrastructure, handling triage, booking, pharmacy and payment outside the hospital. Only ChatGPT Health has been independently tested, and it under-triaged 52 percent of emergencies. Hospitals can’t refuse to send records to an app a patient picks, and HIPAA stops protecting the data once it leaves. The leverage health systems still have sits in procurement contracts, where they can demand local validation and audit rights before signing.

More than 230 million people ask ChatGPT health and wellness questions every week. Not long ago, those conversations ended with advice and a to-do list: find a clinician, book the visit, fill the prescription. Now, some AI platforms are starting to check items off the list on their own, including scheduling appointments and renewing prescriptions.

None of this activity goes through the health system, and a recent Nature Health paper argues that consumer AI has stopped being an information tool and started becoming infrastructure. OpenAI, Amazon, Anthropic, and Ant Group are wiring their AI chatbots into medical records, appointment booking, pharmacy fulfillment, payment, and clinical workflow, and the result runs outside any single institution.

Consumer AI platforms triage patients with almost no independent testing

A platform triages the patient before the hospital sees them, and how often it gets that wrong is mostly unknown. Only one of the four platforms, ChatGPT Health, has faced an independent evaluation. In that test of clinician-written vignettes, it under-triaged 52 percent of emergencies. The authors found no independent evaluations for Amazon Health AI, Afu, or Claude for Healthcare. The companies describe internal testing, but none has published a product-specific number.

Each platform differs in how far downstream it can reach:

• ChatGPT Health offers an optional link to a patient’s EHR and has announced no booking or fulfillment function.
• Amazon Health AI sends questions to One Medical for booking and Amazon Pharmacy for renewals while drawing on the customer’s purchase history. The platform giving the advice also owns the clinic and pharmacy it recommends.
• Afu is Ant Group’s app in China. It is Amazon’s model built out all the way, with one company running triage, referral, pharmacy, and insurance payment.

The authors are careful to say that reach cuts both ways. What matters is whether a platform changes what a patient does next and whether that patient finishes the care they started. A platform that takes a symptom, books a visit, and fills a prescription in one place removes the handoffs where patients quietly drop out of fragmented care.

Hospitals must send patient records to the AI apps patients choose

A hospital might assume it decides whether these apps get its patients’ data. It doesn’t. HHS guidance says a provider generally can’t refuse to send a patient’s electronic record to an app the patient chooses, even over concerns about how the app will use or secure it. Once an outside app receives the data, HIPAA no longer protects it. Under the federal information blocking rules, providers may tell patients about an app’s privacy and security risks, if they treat every app the same way.

What the platforms receive is a portal subset and not the full chart, which the authors call often inadequate for clinical decisions.

A health system can still ask pointed questions about its own side of the exchange:

• What did the patient’s platform actually see, and do our clinicians know what it didn’t?
• Do we review app privacy policies and tell patients where their data goes?
• What happens at intake when a patient arrives already triaged by a platform that missed half of emergencies in testing?
• Is our own portal and scheduling easier to use than the platform’s?

Patient-facing AI skips the hospital reviews that governed Epic’s rise

When Epic became the plumbing of American healthcare, it did so inside institutions. The authors point out that hospital committees, informaticists, and clinicians sat between the software and the patient, and that the rollout took decades. The consumer platforms reached comparable or greater scale in months, and they reach patients directly with no such checkpoint. This time the hospital is the thing being routed around.

Procurement contracts give hospitals leverage over AI validation

If a hospital has little say over the consumer route, the say it does have is worth more, and it lives in the contract. Claude for Healthcare comes in through the back office, via an enterprise subscription and API, with ambient documentation, ICD coding, prior authorization review, claims appeals, and message triage among its functions, plus TEFCA record connections. The authors say tools at this layer could expand clinical capacity and improve coding quality “if validated and adapted.”

The authors treat procurement as the place to enforce validation, above all for health systems short on resources, and a 2025 Health Affairs study shows how often hospitals skip that testing. Among U.S. hospitals using predictive models, 79 percent used models from their EHR vendor, 61 percent tested them for accuracy on local data, and 44 percent tested them for bias.

A hospital can ask its questions before it signs:

• What validation are we giving up for the integration, and is that tradeoff written down?
• Are local validation and equity checks in the contract as acceptance tests before scale?
• Does the contract cover the full interaction trace, grant audit rights, and bar secondary use of payment and insurance records for pricing or risk stratification?
• If we leave, what happens to our data and the workflows built on it?

The authors close by arguing, with resource-limited health systems most in mind, that the question is no longer whether patients will use these tools, but on whose terms. The paper traces the platforms’ pull to habit and the convenience of using accounts patients already have. Every handoff a hospital removes from its patients’ care journeys is one less reason for a patient to start somewhere else.