Decision support tools under scrutiny

The Office of the National Coordinator (ONC) proposed a rule that would make broad changes to the Health IT Certification Program called the “Health Data, Technology, and Interoperability: Certification Program Updates, Algorithm Transparency, and Information Sharing,” or, for short, HTI-1. 

The proposed rule presents changes to many different aspects of the Health IT Certification process, including reworking information blocking rules, implementing an EHR reporting program, and creating higher standards for interoperability and health IT functionality.  

One of the most extensive modifications applies to the existing “clinical decision support” category, which hasn’t been updated since 2012, to regulate how healthcare providers use machine learning, artificial intelligence (AI), or predictive algorithms or models in their clinical practice.  

“[Healthcare providers] simply don’t have information to trust some of the predictive algorithms that are being potentially applied for patient care,” said Jeff Smith, Deputy-Director of Certification and Testing at the ONC during an information session on HTI-1.  

“We really hope that our policies will create a level playing field of consistently available information that will help users be able to determine the difference between poor and high quality predictive models.” 

The first change is to rename “clinical decision support” to “decision support intervention” (DSI) to encompass the array of predictive tools that are being created for healthcare, ranging from administrative to clinical that help healthcare providers make clinical evaluations or decisions.  

As DSI, they would be subject to real world testing.   

The primary goals of the proposed changes are to: 

•  Develop transparency on how decision support intervention tools are designed, trained, tested and used 
• Increase trustworthiness by making available how certified Health IT developers oversee the use and application of predictive decision support intervention tools
• Make information on predictive decision support intervention tools consistently available to allow users to judge the quality of predictive decision support intervention tools to determine whether their suggestions are fair, appropriate, valid, effective and safe (FAVES) 
• Advance health equity by addressing algorithmic bias in predictive decision support intervention tools

ONC strategies to build trust in predictive DSIs

Source attribute requirements 

Health IT Modules that use or interface with predictive decision support intervention tools would be required to provide additional information as source attributes so users can determine the quality of predictive decision support intervention tools and whether and how to use its recommendations.  

The Health IT Modules must also let users know when race, ethnicity, social determinants of health, and other data relevant to health equity are used by a decision support intervention tool. 

“We hope to improve how these models are used for individual patients but also for patient populations which will hopefully start to get at some of the challenges we’ve seen with bias and health disparities,” Smith adds.  

Overall, the modules must give users answers to these four basic questions:  

• What data was used to train the predictive decision support intervention tool? 
• How should the predictive decision support intervention tool be used, updated, and maintained
• How does the predictive decision support intervention tool perform using validity and fairness metrics in testing and in local data, if available?  

Developers of certified health IT would be required to employ or engage in Intervention Risk Management practices, including “risk analysis,” “risk mitigation,” and “governance.” A summary of their risk management plan must be made publicly available and updated annually.  

Real world testing with the flexibility to change 

To provide flexibility, Health IT Modules must allow users to author and edit source attributes to customize predictive decision support intervention tools to their needs. Striving for quality improvement, Health IT Modules must allow users to provide feedback and make feedback data exportable.  

The proposed HTI-1 rule aims to optimize the use of predictive decision support intervention tools in healthcare by improving transparency, enhancing trustworthiness, maintaining consistency, and advancing health equity in predictive DSI tools.  

Public comment period is open until June 20, 2023.