Equity must be an intentional component of AI governance

Artificial intelligence has the potential to either reduce health disparities or amplify them, depending on how it is designed and deployed in the real-world healthcare environment.   

Providers would universally prefer the former, of course.    

But with the rapid pace of AI innovation creating a massive lag between adoption and governance, insufficient guardrails around health equity may mean that organizations are unintentionally baking bias into a system that’s still in flux.   

It’s a risk that the health system simply cannot afford to take, especially as AI catalyzes what is basically a wholesale rethink of the clinical decision-making process, says Katherine Eisenberg, MD, PhD, FAAFP, a practicing family medicine physician at the University of Rochester Medical Center, a CHAI workgroup member, and clinical decision support development expert at EBSCO’s Dyna AI. 

“AI tools are clearly filling important needs.  That’s obvious from how quickly they’re being adopted,” she told Digital Health Insights.  “But the scale and speed of uptake means we’re also conducting a large experiment in equity and clinical decision-making.” 

“There’s no rulebook yet for how we approach health equity and bias mitigation in AI, but we need to be trying to write those guidelines,” she continued.  “An intentional, proactive, good-faith effort will go a really long way toward developing robust governance that keeps AI moving in the right direction.” 

Recognizing the subtle signals of AI bias in the clinical setting

Bias in clinical decision-making isn’t always obvious.  It can find its way into an AI system at every stage of the digital life cycle, from conceptual design and training data collection to algorithm development, validation, and clinical implementation.   

“Imagine asking an AI system about a patient with a racing heart, sweating, and general malaise,” Eisenberg explained. “If I say it’s a male, the system will likely suggest a heart attack as the most likely diagnosis. But if I simply change the patient’s gender to female, suddenly anxiety becomes the leading diagnosis, even though there isn’t enough clinical information to justify those different conclusions.”  

“That’s the kind of subtle bias that can creep into AI responses. You often have to compare cases side by side to recognize it, which means we have to actively look for these issues rather than assume we’ll notice them.” 

The challenge is compounded when training data is biased to begin with, she added.  Research has shown that AI training datasets often exhibit gender and racial bias, a perpetuation of long-standing biases in clinical trial data and other medical literature upon which many medical AI systems are built.  

“If there are preexisting gaps in the medical literature or certain groups are underrepresented in clinical trials, you can only make up so much of that with a policy in your AI tool,” Eisenberg stated.  “But the important thing is that we do our best to try.” 

“It’s entirely possible that we’re perpetuating certain biases simply because there hasn’t been enough time to study them. That’s why being intentional and proactive is so important.”  

Pulling the right levers to put equity front and center

Equity gaps in the AI environment can surface in all sorts of ways, not just on the patient level.  For example, disparities in health system resources and technical skills can lead to different degrees of AI validation in the local setting, Eisenberg said, which in turn can leave some systems more vulnerable to hidden biases than others. 

“One of the industry’s biggest unresolved challenges is figuring out how to take tools that inevitably reflect their training data and apply them appropriately in local settings to meet the needs of specific populations,” she said.  

“However, systems with fewer resources often have to rely on out-of-the-box tools because they simply don’t have the staff or technical expertise. I don’t think we have good data yet on what that looks like in practice, but I think it’s a real risk for communities being served by resource-constrained health systems.  It’s a double whammy for them, because many of those populations are the ones that are already underrepresented in the literature to begin with.” 

But healthcare organizations, even those with limited resources, do have power in shaping the AI equity conversation.  “Buyers have a tremendous amount of influence over what the industry produces,” Eisenberg pointed out. “The questions they ask shape vendor priorities.” 

“I’d absolutely encourage buyers to keep questions about equity, bias, and local validation high on the list when talking with a potential partner. It doesn’t actually take a lot to build an effective equity evaluation, but it does have to be a priority within your procurement process.” 

Organizations can also seek out a growing number of third-party collaborators who are working across the vendor, provider, and policymaker communities to establish robust, accessible methodologies for operationalizing governance.   

“The Coalition for Health AI (CHAI) has played an important community-building and advisory role,” she noted. “There are implementation efforts underway through organizations like the Digital Medicine Society, and many others are trying to help organizations understand how to operationalize AI governance.” 

“What we’re wrestling with as an industry is what the roles should be, who pays for them, and how we can keep up with technology that’s evolving so quickly.  Those are questions that everyone needs to participate in answering so that we can create an ecosystem that works for all its members.” 

Developing a future of AI governance that supports health equity

AI is now maturing to the point where the health system is redesigning care around it.  It’s a promising prospect for clinicians and for patients, but only if AI tools bring benefits that can be equitably applied to all populations.  

“There’s a tremendous opportunity to use AI not only to deliver better information at the right moment, but also to identify who needs preventive care, transportation assistance, or other social supports and connect them with those services more effectively,” said Eisenberg. 

“Our goal as leaders in AI governance and digital development must be to share strategies for identifying bias and reducing the impact of bias on the information that is being used to make decisions.  If we can assist the industry with keeping the concepts of bias and health equity front and center during the AI adoption process, we can set ourselves up for greater success with harnessing all of the benefits AI can bring to the care delivery process.” 

Jennifer Bresnick is a journalist and freelance content creator with a decade of experience in the health IT industry.  Her work has focused on leveraging innovative technology tools to create value, improve health equity, and achieve the promises of the learning health system.  She can be reached at [email protected].