NAACP releases blueprint to tackle bias in healthcare AI

In 2019, a research team led by Ziad Obermeyer at UC Berkeley gained access to a widely used commercial algorithm designed to help hospitals identify “high-risk” patients—those who would benefit from additional nurse visits, care coordination, and chronic disease management.

When the researchers examined how the algorithm performed across racial groups, a troubling pattern emerged. Black patients carried significantly more disease burden than White patients with the same risk scores. At identical algorithm outputs, Black patients had 26% more chronic conditions, including higher rates of hypertension, diabetes, anemia, and kidney disease.

The algorithm wasn’t broken. It was doing exactly what it had been built to do.

The system used healthcare spending as a proxy for healthcare needs, but it failed to account for well-documented disparities in access to care. Because Black patients face barriers such as insurance gaps, geographic distance, discrimination, and justified medical mistrust, they often spend less on healthcare even when seriously ill. The algorithm learned this pattern and treated lower spending as lower need. Structural inequality was absorbed into the model and reproduced as clinical judgment.

A new white paper from the NAACP and Sanofi argues that the Obermeyer case exposes a core truth about healthcare AI: these systems are not just technical tools. They are socio-technical systems embedded in histories of inequality. As such, they demand governance approaches grounded in transparency, accountability, and community power. Without that, the paper warns, AI will automate inequality at scale.

Go beyond the hype—learn how healthcare organizations are operationalizing AI today at ViVE. AI Zone @ViVE 2026.

How bias enters the pipeline

The report describes several mechanisms through which this “automation of inequality” occurs. One is unrepresentative training data. Dermatology provides a clear example. Many image datasets used to train diagnostic AI systems are drawn from academic medical centers serving predominantly White, insured populations. Analyses show that darker skin tones make up less than 15% of these datasets, despite representing roughly 40% of the U.S. population. When evaluated on Black patients, AI systems trained on this data perform 20% to 50% worse for certain conditions. The technology is not learning dermatology broadly—it is learning a narrow version shaped by who was included in the data.

Bias also enters through the medical devices that supply data to AI systems. During the COVID-19 pandemic, pulse oximeters became essential tools for assessing oxygen levels. Research later published in JAMA Internal Medicine found that these devices systematically overestimated oxygen saturation in Black patients, with error rates up to three times higher than those seen in White patients. Because pulse oximeters rely on light absorption through skin, and had been calibrated primarily on lighter skin tones, Black patients with dangerously low oxygen levels were more likely to appear stable. As a result, they were less likely to receive supplemental oxygen or intensive care—even when their true oxygen levels matched those of White patients. This flaw had gone largely unexamined for decades.

The white paper also highlights what it calls a “capacity chasm.” The healthcare organizations serving the most vulnerable populations—community health centers, rural hospitals, and urban safety-net providers—are often the least equipped to evaluate or challenge AI systems. These providers typically lack the technical infrastructure, data science expertise, and financial resources needed to validate commercial algorithms against their patient populations or to build representative datasets of their own.

In a survey of 230 safety-net providers conducted by University of Texas researchers, 38.8% saw potential for AI to improve administrative efficiency. At the same time, 31.9% cited data privacy concerns and 17.3% pointed to lack of funding as major barriers to implementation. AI tools are frequently developed without these communities at the table, then deployed in settings where their limitations carry the greatest risk.

Giving communities real power over AI

To address these issues, the NAACP task force proposes a three-layer governance framework spanning ethical, organizational, and operational domains. One of its most striking recommendations is at the organizational level: Community Advisory Boards should have real authority, including the power to pause or shut down AI systems if concerns arise. This goes beyond consultation or feedback. It represents a shift in who holds decision-making power.

The framework also calls for Data Governance Councils to maintain clear records of where training data comes from and which populations it represents. It supports mandatory Model Cards and Datasheets for Datasets that document algorithm performance and limitations across demographic groups. In addition, it proposes quarterly AI Equity Reports—plain-language disclosures that allow patients, clinicians, and community members to understand how systems perform in practice.

Maternal health serves as a case study for how these principles could be applied. Wearable technologies can track heart rate variability, sleep, and stress throughout pregnancy, potentially enabling earlier detection of complications. This is especially significant given that Black women experience maternal mortality rates roughly four times higher than those of White women. But these tools also involve sensitive data collection, including audio, location, and biometric monitoring, in communities with long-standing reasons to distrust medical surveillance.

A review of maternal health AI studies found that Black and Hispanic women were consistently underrepresented in validation trials. As a result, tools designed to identify early warning signs were least reliable for the populations most at risk. The NAACP argues that ethical deployment requires clear data use agreements, routine equity impact assessments both before and after deployment, and meaningful involvement of community-based providers—not just as users, but as co-designers with lived expertise.

What synthetic data can and cannot fix

The paper identifies synthetic data as one approach that would support equity. Rather than relying on real patient records, synthetic data uses computational models to generate artificial datasets that reflect real-world variation, such as differences in image quality, caregiving environments, and family structures. In one study, dermatologists rated computer-generated skin images as realistic about 90% of the time, particularly when those images reflected the lighting and conditions typical of primary care settings rather than specialty clinics.

For under-resourced health systems, synthetic data offers a way to test and validate AI tools without compromising patient privacy or requiring extensive infrastructure. But the report cautions that synthetic data is not a shortcut to fairness. If generated without input from affected communities, it can reinforce inaccurate assumptions and cultural blind spots. You cannot simulate what you have never measured or understood. Equity still depends on inclusive design from the outset.

Underlying all of these concerns is a fundamental question about data ownership and value. Patient data—medical histories, biometric signals, behavioral patterns—fuels the AI economy. Marginalized communities contribute disproportionately to this data through public hospitals, safety-net systems, and Medicaid programs, yet often see the fewest benefits from the resulting technologies. The NAACP frames this as a data rights issue, linking AI governance to broader debates about who profits from data and how economic value is distributed.

AI is creating two classes of healthcare workers

The workforce implications add urgency. The Bureau of Labor Statistics projects that healthcare and social assistance will grow faster than any other sector through 2034. Many of these jobs—nurse practitioners, physician assistants, community health workers—serve safety-net populations. The white paper warns of an emerging “AI divide,” in which clinicians with access to training and infrastructure gain new diagnostic and preventive capabilities, while others are left behind. Without intervention, AI could deepen a two-tier healthcare system.

The environmental cost of AI

Lastly, the report turns to infrastructure. Data centers that power AI development consume enormous amounts of water and electricity, with major technology companies planning hundreds of billions of dollars in new investments. Communities hosting these facilities report strain on local resources. The NAACP argues that environmental impact assessments should be part of AI governance, recognizing that the physical footprint of AI also raises questions of justice.

The algorithm Obermeyer studied did not fail by accident. It reflected the priorities and assumptions of a healthcare system where access is unequal. The question, the NAACP argues, is whether society will redesign that system as it automates it—or simply encode inequality faster. AI will shape the future of healthcare. But the future of healthcare AI is still a choice.