It’s 2026. Do you know where your AI is?

If there’s one rule that governs safety and success across the entire healthcare delivery process, it’s “know where your stuff is.”  Whether it’s controlled substances in drug lockers, clamps in the OR, dollars in accounts receivable, or even actual patients in their beds, keeping track of what should be where is a fundamental competency for everyone across the care continuum. 

It’s just as critical in the IT department. However, as artificial intelligence becomes increasingly embedded across clinical and operational workflows, many organizations only have a partial understanding of where AI is really being used – and what risks those tools can introduce if they’re not properly accounted for. 

“Shadow AI” is becoming a growing problem, for example, as individual teams and staff members start exploring the wide array of AI tools to make their specific workflows easier. Around 40% of healthcare organizations have encountered unauthorized AI use, according to recent data, often due to unclear understanding of policies around appropriate AI adoption.  

The challenge is likely to intensify as piecemeal AI adoption collides with a rapidly growing population of non-human identities, including AI agents, service accounts, APIs, and machine credentials. 

To avoid the many risks of unknown tools and non-human actors entering a system that is already incredibly vulnerable to cyberattacks, AI inventory management must become a foundational component of AI governance. 

New guidance from the Health Sector Coordinating Council’s Cybersecurity Working Group reinforces the critical importance of AI inventory management and offers a framework for how organizations can appropriately identify systems, vendors, model owners, and risk classifications before those technologies create security, privacy, or patient safety issues. 

Success starts with developing an AI governance framework that prioritizes visibility and accountability, because it’s simply not possible to adequately address the risks of something that nobody knows exists.  

The challenge of identifying AI in a highly complex digital environment

With almost every digital health vendor integrating AI into their products in some manner, it’s increasingly difficult to keep track of where AI capabilities are entering the IT ecosystem, the report notes. 

“AI enablement may occur without a discrete procurement or onboarding event, creating material governance, compliance, cybersecurity, and patient safety risks if not properly controlled,” the authors caution. “This includes AI embedded in clinical devices managed by clinical engineering, cloud hosted AI services, and new AI features enabled via automatic updates, firmware upgrades, or feature releases.” 

Leaders must account for this “dynamic introduction” of AI capabilities through robust IT Asset Management (ITAM) practices, including tracking upgrade cycles of new and existing tools, registering and assigning ownership of new assets, and monitoring AI enabled tools throughout their lifecycle to appropriately retire unused, duplicate, or end-of-life assets. 

The report suggests that organizations develop a comprehensive AI catalog that incorporates “all AI use cases, data dependencies, model owners, and business sponsors across the organization.”   

While this may appear to be a tall order, given the scale and speed of AI adoption, a strong focus on visibility, traceability, and accountability may be crucial for preventing the many vulnerabilities associated with shadow AI, both from a patient safety perspective and through the lens of organizational cybersecurity.  

Accounting for new risks and attack vectors associated with AI

On the cybersecurity front, AI introduces a number of new attack surfaces that are already being exploited by cybercriminals, who are using creative tactics such as prompt injection and data poisoning to gain access to sensitive systems. 

Combatting these attacks requires developing defensive architecture that includes strong identity management, data loss prevention, penetration testing, and thorough patching and updating protocols. 

For instance, the Health Sector Coordinating Council also recommends “AI red teaming” that is distinct from traditional penetration testing that focuses specifically on issues such as prompt injection resistance, jailbreak resistance, and privilege escalation, all of which require different approaches than other types of IT systems. 

But it goes without saying that none of these activities can have any effect on AI that isn’t on the list of existing assets, further reinforcing the need for a complete, accurate, and up-to-date inventory of AI tools. 

Turning inventory into action to enhance security and effectiveness of AI

Ultimately, AI governance should adopt the same rigor, discipline, and continuous surveillance that healthcare organizations already apply to other critical assets in the clinical and operational spheres. 

Every AI asset should have an owner, a risk classification, and a documented place within the organization’s inventory and oversight processes – and these factors should be reviewed and revised as appropriate on an ongoing basis.  

That visibility must also extend beyond the models themselves to include the growing ecosystem of service accounts, API connections, machine identities, and agents that support AI-enabled workflows. 

Without that foundation, even the most sophisticated cybersecurity and governance strategies will fall short of offering the real protections necessary to monitor and manage the highly complex, rapidly changing AI environment. 

Jennifer Bresnick is a journalist and freelance content creator with a decade of experience in the health IT industry.  Her work has focused on leveraging innovative technology tools to create value, improve health equity, and achieve the promises of the learning health system.  She can be reached at [email protected].