When AI explains itself, does it help or hurt clinical decision-making?

It looks like AI for clinical decision-making might be caught between a rock and a hard place.  Industry observers have continuously warned about AI’s “black box” problem and cautioned clinical users against trusting recommendations blindly.  But new research involving both clinicians and laypeople indicates that adding explanations to an AI recommendation might actually do more harm than good, in some situations. 

Published in Nature Medicine this month, the study enlisted both primary care physicians and non-clinical participants to look at images of skin lesions and make a diagnostic decision. They were given a variety of types of information to help, including a basic AI prediction without any additional explanation, LLM-generated text explanations in addition to the prediction, or example images of similar skin conditions. 

The results show the strikingly persuasive impact of authoritative-sounding AI information, and the challenge of hitting the right balance between being helpfully explicable and unintentionally creating automation bias.  

Non-clinical folk are easily swayed when AI sounds sure of itself

For the lay participants, LLM-generated explanations had a significant influence on their ultimate decision-making – even when the explanations were wrong. When the AI diagnosis was correct, the explanations improved participants’ accuracy by 13.4%, more than basic predictions or examples of similar images.  

But when the AI was wrong, LLM explanations caused the largest decline in accuracy: a 21.1% drop in accuracy compared with a 14.6% reduction when just given the AI answer alone.  

In other words, the mere presence of the text-based explanations amplified reliance on the AI in both directions, rather than serving as a tool to help participants evaluate whether or not the suggestion was correct.   

The explanations also had an impact on the confidence of the participants. Participants exposed to explanations often became more confident in the accuracy of their answer regardless of whether the AI was correct. When the AI made an incorrect recommendation, the LLM explanation frequently boosted the participants’ confidence in the wrong answer, suggesting that participants often found plausible-sounding explanations convincing even when those explanations were intentionally low quality. 

The authors note that the explanations often offered plausible-sounding reasoning based on ambiguous or visually unclear dermatological features, which were convincing enough for someone without pre-existing clinical knowledge to believe.  

Clinical judgement says steadier in the face of AI decision support

On the bright side, trained clinicians were much more resistant to being persuaded by the AI explanations if they did not match their own clinical judgment, and were more likely to use the explanations as a tool for critically evaluating their decision-making process.   

Using AI to inform their decisions made them more accurate overall, but the LLM explanations were not a particularly helpful format for improving diagnostic accuracy, the study found. The explanations generated a 17.7% improvement versus a 25.8% improvement with the basic AI prediction alone. The authors note that this difference is not considered statistically significant, indicating that LLM-generated explanations did not meaningfully improve diagnostic accuracy beyond the AI prediction alone. 

However, when physicians received LLM-generated explanations, their confidence became better aligned with whether they were actually correct. They weren’t necessarily more accurate, but they had a better sense of when to be confident and when to be cautious, suggesting that the explanations may be helpful when users have the background knowledge to critically evaluate their content. 

The next ethical AI design challenge: transparency versus persuasiveness

The results around the lay participants deserve extra scrutiny, particularly as the use of AI for initial diagnostic and decision-making support skyrockets among everyday users. Many patients are using AI chatbots to get physical or mental health advice, with a KFF poll recently finding that one in three people use AI for this purpose.   

And with the survey revealing heavier users more likely to trust AI output than people who don’t typically rely on AI, it’s now more important than ever that policymakers, developers, and clinicians fully understand the impact of AI information on how people are choosing to engage with care. 

If the simple presence of plausible explanations is interpreted as evidence of correctness rather than as supplementary information to support critical thinking, then that adds a whole new twist to the “black box” debate.   

The findings suggest that explicability needs to be viewed as more than just a technical challenge. It’s not about whether or not an AI tool is able to produce a viable explanation of its internal workings, but about how that explanation influences the way specific audiences make decisions.  

To address these issues, developers, academic researchers, and policymakers will need to dive deeper into how presenting specific pieces of information to unique audiences impacts trust, persuasiveness, and actionability. The goal should not be simply to provide the information, but to present the right information in the right context to encourage appropriate critical thinking. 

Jennifer Bresnick is a journalist and freelance content creator with a decade of experience in the health IT industry.  Her work has focused on leveraging innovative technology tools to create value, improve health equity, and achieve the promises of the learning health system.  She can be reached at [email protected].