KPIs for compassionate healthcare technology

After implementing ambient AI for clinical documentation, physicians at Akron Children’s Hospital were saving 90 minutes per week and burnout scores had dropped by 22 percent.

But proving those softer benefits to hospital executives used to require little more than anecdotal feedback and wishful thinking. Now, a growing number of health systems are developing sophisticated measurement frameworks to quantify what they call “compassionate ROI” to track not just whether new technology saves time and money, but whether it makes clinicians and patients happier.

The push comes as hospitals face mounting pressure to justify AI and digital health investments that promise to ease clinician burnout while improving care quality. Traditional metrics like system uptime and user adoption rates no longer tell the whole story, according to healthcare IT leaders who gathered at CHIME’s inaugural CompassionIT Summit last week. 

“We are creating overbearing digital work for humans who, at times, have difficulty managing it,” said Dr. Alan Miller, chief medical information officer at Nuvance Health, during the event. “The question is: How do we measure whether we’re actually helping or hurting?”

The new scorecard

Health systems are borrowing measurement tools from aerospace and psychology to assess technology’s impact on clinician wellbeing. Akron Children’s Hospital has started using the NASA Task Load Index, originally developed to measure pilot workload, to evaluate how mentally demanding their electronic health record systems are for doctors and nurses.

On a scale where scores above 10 indicate high cognitive burden, physicians’ ratings for “hard work” dropped from 13.8 to 6.8 after implementing ambient AI documentation tools. Mental demand scores fell from 12.8 to 5.9.

“Any difference of two points is a huge game changer,” said Harun Rashid, CIO at Akron Children’s. “We can now show executives that we’re not just saving time, we’re reducing the mental load on our clinicians.”

Denver Health has taken a similar approach, using validated burnout assessment tools to track changes over time. Their pilot group saw burnout scores drop by more than 50% after implementing ambient AI for documentation.

From pajama time to patient satisfaction

The new measurement frameworks track metrics that would have seemed foreign to hospital administrators just a few years ago. “Pajama time” — the hours clinicians spend on documentation at home after regular work hours — has become a key performance indicator. Patient experience scores are being analyzed as leading indicators of technology success, not just lagging outcomes.

Dr. Daniel Kortsch from Denver Health said his team has seen patient satisfaction increase by at least 9% following AI implementation, improvements that flow through to publicly reported quality metrics and reimbursement rates.

“Although ambient AI really can improve productivity, which is historically where most people go with this, in my mind that is really the short game,” Kortsch said. “The long game is recognizing that there are softer things that are still calculable.”

Find out more information about CHIME’s CompassionIT Summit here, or watch the full recording here.