Human vs. AI: Clinicians to test algorithms in real-world settings

Ever wanted to get your hands on the latest artificial intelligence tools to see what they can really do – and really can’t do? Mass General Brigham (MGB) is giving some lucky clinicians the chance to dig into some of the most promising, late breaking AI solutions in simulated real-world settings so they can collect feedback from actual users with experience in direct patient care. 

The Healthcare AI Challenge, hosted by MGB, will convene a variety of leading health systems, academic organizations, and professional societies to participate in an interactive series of events to “crowdsource” input from the healthcare community. Faculty with relevant medical credentials, as well as members of participating professional societies, will get access to the testing environment, where they can provide their opinions on select tools. 

“The velocity of AI innovations and breadth of their healthcare applications continues to increase. This unprecedented growth leaves clinicians struggling to determine the effectiveness of these innovations in safely delivering value to healthcare providers and our patients,” explained Keith Dreyer, DO, PhD, Chief Data Science Officer at Mass General Brigham, and leader of Mass General Brigham AI, the healthcare system’s AI business.  

“The Healthcare AI Challenge is a collective response to the complexities involved in advancing the responsible development and use of AI in healthcare. This new approach strives to put clinicians in the driver’s seat, allowing them to evaluate the utility of different AI technologies and ultimately, determine which solutions have the greatest promise to advance patient care.” 

The first event will focus on radiology, a robust and relatively mature area of AI development.  

“Medical imaging provides many types of data, and up to 95% of healthcare data collected is unstructured, non-text data,” said Richard Bruce, MD, associate professor of radiology and the vice chair of informatics at the University of Wisconsin School of Medicine and Public Health, a founding participant organization.   

“AI has the potential to interpret and distill that data at a new scale and speed, but what we need is the ability to quickly test and compare different AI solutions. The Healthcare AI Challenge will offer a platform to evaluate and compare tools across various clinical situations.”  

Clinical experts who are part of the collaborative can log into the portal and choose one of the available models to assess. An image interpretation challenge, for example, may include questions about draft report generation, key findings, differential diagnosis, among others, the press release says. The participant then rates the clinical skill level of the foundation models’ responses, which will contribute to ranking the model. The rankings will provide industry stakeholders with objective, transparent, and knowledgeable assessments of AI performance in key areas of care, which will in turn foster more precise, effective, accurate, and trustworthy model development in the future. 

Founding members of the Healthcare AI Challenge include Emory Healthcare; the Department of Radiology at the University of Wisconsin School of Medicine and Public Health; and the Department of Radiology at the University of Washington School of Medicine. The American College of Radiology (ACR), a professional medical society representing radiologists, has also joined, enabling its 42,000 members to participate. MGB will announce and onboard additional institutions as the initiative progresses.  

The challenge represents a unique way of addressing the challenges of incorporating experienced clinical viewpoints into the AI validation process. Often, AI models developed in-house at health system, or deployed in only one primary pilot environment, may suffer from unconscious biases due to limited data or institution-specific habits, guidelines, and best practices.   

Allowing clinicians from a wider array of backgrounds and experience levels to test the same models could make it easier for developers to understand how their tools will function in the wild, and give them a chance to make corrections and improvements to account for variation in real-world utilization.  

Clinicians at member organizations can sign up for challenges here, while observers can follow along with the leaderboard results at healthcareaichallenge.org. 

Jennifer Bresnick is a journalist and freelance content creator with a decade of experience in the health IT industry.  Her work has focused on leveraging innovative technology tools to create value, improve health equity, and achieve the promises of the learning health system.  She can be reached at jennifer@inklesscreative.com.