To measure the real value of AI, start looking beyond the dollar signs

What does a successful AI implementation look like?  It’s a very basic question that healthcare leaders are still struggling to answer, even after already pouring billions of dollars into AI-enabled tools and workflows. 

How can organizations accurately measure the value of their digital transformation efforts?  Is it about comparing the implementation costs to additional revenue generated?  Improvements in clinical outcome metrics?  Higher performance on patient experience surveys?  Or maybe it’s a mix of all of these – and more?  

With a new framework pre-published in NPJ Digital Medicine, a pair of emergency physicians and researchers at UVA and Clemson University are urging executive leaders to embrace the latter.  A holistic and multi-dimensional approach to defining and measuring the value of AI tools can help healthcare organizations craft a more relevant and informative definition of ROI for the modern digital environment, assert the authors. 

“Hospitals are seeing a huge number of new AI tools marketed to improve healthcare. The challenge is to figure out which ones actually will,” said co-author Arwen B.L. Declan, MD, PhD, clinical assistant professor in Clemson University’s School of Health Research. “That requires weighing an AI tool’s impact across clinical, operational and financial dimensions, while keeping patient care at the center of every decision.”  

Defining the “total mission value” of AI implementation decisions

The proposed framework encourages healthcare leaders to view AI adoption through a lens of “total mission value,” or TMV, which combines traditional financial ROI metrics with the organization’s mission-driven goals.   

Rather than asking only whether an AI system saves money or generates revenue in specific use cases, this approach questions whether a tool advances the broader purposes of healthcare, including patient care, workforce well-being, education, and research, in addition to remaining economically sustainable.   

TMV isn’t designed to fully replace cost-effectiveness or health economic analysis, but rather to offer a more comprehensive approach to AI adoption and governance decisions. 

“AI governance should integrate mission-driven values with economic metrics, because evaluating AI based primarily on cost risks separating technology decisions from healthcare’s core purpose,” wrote Declan and co-author R. Andrew Taylor, MD, MHS, vice chair of research and innovation for the University of Virginia School of Medicine’s Department of Emergency Medicine.  

Five domains for measuring the total mission value of AI 

The framework includes five interconnected domains for assessing the value of their new tools.  

Patient care sits at the top of the pyramid.  To quantify the impact of AI in this area, leaders should consider examining metrics tied to clinical outcomes, patient satisfaction and engagement, and population health outcomes.  Paying particular attention to how AI tools increase or reduce health disparities will be crucial for accurately evaluating success.  

Next is the workforce experience.  Much has been made about minutes saved with tools like ambient listening, but productivity isn’t the only metric to observe.  The team suggests including other measures of workforce experience, such as assessing burnout rates and cognitive burdens, staff retention rates, after-hours charting time, and overall engagement and job satisfaction.  

The third bucket, operations, is at the heart of traditional AI governance planning, and can complement the previous two areas.  By asking questions about how much AI has improved the workflow and reduced risk while maintaining quality and performing in an equitable manner, leaders can better assess its impact on experiences and outcomes for both patients and staff members.  Examples of key performance indicators in this area may include EHR use time, note completion rates, and other measures of workflow efficiency.  

Economic evaluation follows swiftly after, and includes foundational assessments of financial performance, patient costs, revenue, billing integrity, and capital allocation.  The authors reiterate that over-indexing on financial ROI alone can create conflicts with an organization’s mission statement and values, which is why this domain is only one of the five areas to consider when making AI adoption decisions. 

Last is the category of educational and research impact.  Instead of asking only whether AI improves workflows and outcomes today, leaders should consider the long-term effects of AI on the learning health system.  Does an AI tool strengthen evidence-based practice, foster innovation, and generate new knowledge that can help future patients and providers?  Rising concerns about workforce “deskilling” and the influence of AI on the human aspect of the patient-provider relationship make this domain more important than ever. 

The authors note that instead of considering measurement of health equity as its own task, equity “serves as the essential, unifying foundation that connects and informs all other domains.”  By embedding equity into each area, leaders can be sure to maintain an appropriate focus on the topic at all times. 

Leveraging a TMV approach to make more informed AI decision

With the rapid proliferation of AI tools and increasing pressure on healthcare organizations to keep pace with their peers, it’s only getting more difficult for leaders to find and implement effective, high-quality AI products to help them achieve their goals.   

Health systems have little room for expensive and time-consuming failures, creating even greater incentives to look beyond vendor promises and toward how a tool will really function in the real-world environment. 

A broader evaluation framework helps leaders evaluate AI investments according to the specific challenges that affect their environments – and the specific outcomes that matter most to their own missions – rather than relying on limited financial metrics alone, which may overlook important effects on patient care, workforce well-being, and operational efficiency.  

Instead of asking only whether an AI tool pays for itself, leaders are more likely to have robust discussions about which stakeholders really benefit, what trade-offs and caveats may exist, and whether a technology still advances the organization’s goals as a whole.   

Jennifer Bresnick is a journalist and freelance content creator with a decade of experience in the health IT industry.  Her work has focused on leveraging innovative technology tools to create value, improve health equity, and achieve the promises of the learning health system.  She can be reached at [email protected].