Study of 271 million OpenEvidence queries shows clinicians use AI as a colleague, not a textbook, raising evaluation questions

Key takeaways

A federally funded JAMA study from Harvard Medical School and Brigham and Women’s Hospital analyzed 271 million OpenEvidence queries from more than 950,000 clinicians over two and a half years and found 33.1 percent concerned treatment and management while only 13.9 percent were about diagnosis, with out-of-specialty queries ranging from 35 percent among oncologists to 88 percent among emergency physicians. That pattern conflicts with how clinical AI is evaluated, since a 2026 review found 95 percent of 519 studies tested tools primarily on accuracy, 45 percent used medical licensing exam questions and only 5 percent incorporated real patient data. JAMA’s accompanying editorial calls for evaluations that measure how well platforms support clinicians’ actual information needs, and for further study of how heavy AI use by early-career clinicians affects skill development.

AI has transformed the first generation of CDS tools into supercharged systems that make more clinical knowledge more accessible and sophisticated than ever.

With that power comes risk, however, which is why developers and policymakers have spent a huge amount of effort developing evaluation frameworks to ensure the safety and accuracy of these clinical AI tools as they suggest diagnoses or recommend medications.

But there may be a problem. A new study analyzing more than 270 million AI queries reveals that real-world clinical users aren’t just treating AI decision support like a textbook on steroids.

Instead, they’re viewing clinical AI more as a second opinion from a colleague on treatment and patient management, and they’re often asking about issues that lie outside their direct area of clinical specialty.

And standard safety and accuracy evaluation tools, which tend to focus on how well AI can answer the type of questions found on a medical exam, might not be keeping up. The result may be a growing mismatch between how AI is tested and monitored and how clinicians are really engaging with it in the clinic.

A closer look at real-world patterns of AI use

A team of researchers from Harvard Medical School and Brigham and Women’s Hospital has published data from one of the first large-scale studies of clinical AI use. The federally funded effort, published in JAMA, looked at how more than 950,000 physicians, PAs and NPs used the OpenEvidence platform in practice over a period of two and a half years.

Several of the authors are employees of OpenEvidence, which did not fund the study but did provide the requested de-identified data for analysis.

“AI tools have become integrated into routine clinical practice faster than almost any technology we’ve seen, but until now we’ve had very little data on how clinicians actually use them,” said first author Vishal Patel, HMS clinical fellow in surgery at Brigham and Women’s. “This research gives us an early picture of where these tools fit into clinical decision-making and where we need to study their accuracy and effects on patient care next.”

The study found that 33.1 percent of the 271 million queries concerned treatment and management issues, while just 13.9 percent were primarily about diagnoses. And queries on topics outside the user’s own specialty ranged from 35 percent among oncologists and hematologists to 88 percent among emergency department physicians, indicating that users may be looking for more active guidance about treating and managing patients, not just trying to find out what might be wrong with them.

Other key highlights include:

• Monthly active users increased from 355 in January 2024 to more than 700,000 in June 2026, with average monthly queries per active user rising from 18.0 to 39.5 over the same period.
• Early-career clinicians, including residents and fellows, were more likely to submit queries than more experienced clinicians. Nurse practitioners and physician assistants were more frequent users than physicians as well.
• Physicians specializing in oncology and hematology, internal medicine and family medicine made more queries per user, while physicians from procedural specialties (radiology, ophthalmology, orthopedics) used the platform less often.

Are evaluation frameworks taking these usage patterns into account?

The potential issue is that AI clinical decision support tools have traditionally been evaluated on their medical knowledge.

For example, a 2026 review cites an analysis of 519 studies published between 2022 and 2024 showing that 95 percent of those studies primarily evaluated AI CDS tools on accuracy, while 45 percent tested medical licensing exam questions and 19 percent focused on diagnostic capabilities. Only 5 percent incorporated real patient data.

The problem is that AI tools that can answer cut-and-dried questions with acceptable accuracy might not perform equally well on complex reasoning that involves synthesizing multiple types of actual patient data, the review’s authors contend.

And if evaluation frameworks aren’t designed to gauge how well AI tools perform on those types of tasks, the type that clinicians are increasingly asking of them, then there could be a worrying gap in the industry’s safety and accuracy monitoring capabilities.

Ensuring alignment between evaluation and real-world use

In an editorial accompanying the publication of the OpenEvidence study, the JAMA editorial team notes that the observed query patterns bring attention to the “breadth of knowledge clinicians are required to draw upon in routine clinical practice,” and suggests “that information platform evaluations should address not only whether they can accurately and reliably answer diagnostic or treatment-related questions, but also how well they support clinicians’ information needs in actual care.”

They also highlight the higher AI usage of early-career clinicians, and assert that it will be important to further study how AI use may affect the development of clinical decision-making skills, as well as information retention, especially for those still refining their professional abilities.

Overall, there is a need for additional study into how clinical AI tools are being used in the real world and how well evaluation efforts align with these factors.

“As the body of medical evidence continues to grow, the goal should be more ambitious than offering clinicians easier access to information,” the editorial team concluded. “AI-enabled clinical information platforms should strengthen clinicians’ ability to understand, evaluate and apply evidence in caring for patients.”

Jennifer Bresnick is a journalist and freelance content creator with a decade of experience in the health IT industry.  Her work has focused on leveraging innovative technology tools to create value, improve health equity, and achieve the promises of the learning health system.  She can be reached at [email protected].