How to make sure LLMs aren’t generating memorized outputs

As their name implies, large language models (LLMs) rely on vast data sets to improve accuracy and reduce bias in outputs across multiple use cases. For healthcare, that typically means using deidentified data from electronic health record (EHR) systems to train LLMs. 

In theory, an LLM would generalize the information it’s trained on, drawing on many records to make recommendations and predictions. Two recent research papers, one led by MIT and another by Ohio’s Franklin University, found healthcare LLMs may be prone to memorization, which poses clear data privacy and patient safety risks. Fortunately, both papers also presented advice for mitigating the issue.  

Testing for a model’s memorization tendencies

The challenge: The MIT-led team published its research in November 2025. Their paper determined LLMs are more likely to leak information if the person entering the prompts, whether attacker or clinical user, knows a lot about a specific patient.  

Admittedly, different types of data leakage pose different risks. General demographic information is fairly benign, researchers told MIT News, as is a single clinical data point such as a white blood cell count. (The caveat is whether enough personal identifiers, as defined by HIPAA, are revealed to re-identify the patient.)   

On the other hand, leaking a specific diagnosis or sensitive information is problematic – to a point. In certain situations, researchers said, an individual might need to know enough about a patient, such as the dates and values of multiple lab tests, to extract information from an LLM. In that case, an attacker would have no need to touch a model; they’d know enough about the patient already. 

The solution: To help organizations address the memorization and leakage issue, researchers created several privacy evaluation tests, available on GitHub. The tests “quantify different forms of memorization and assess their implications in clinical settings, distinguishing harmful leakage at the patient level from useful generalization at the population level.” 

Comparing general-purpose and domain-specific LLMs

The challenge: The Franklin University team, with researchers based on three continents, noted LLMs “might inadvertently memorize and regurgitate private information.” Their January 2026 paper indicated memorization is a clear threat to patient privacy and regulatory non-compliance. (Memorization was one of three LLM risks researchers explored; the other two were generating incorrect or unsafe response from errors in prompts and retrieving irrelevant or confidential information from external sources.) 

The paper found ChatGPT 4, the general-purpose LLM that 230 million people globally use to answer healthcare questions, had an exact match rate of 2.4%. That meant, for a given prompt, the model’s output was “an exact replica of a phrase or sentence from the training data.” This compared to rates between 1% and 1.4% for healthcare-specific LLMs such as Med-PaLM, BioBERT, and ClinicalBERT – the latter of which also outperforms ChatGPT 4 in diagnostic accuracy, University of Utah researchers found. 

The solution: Using LLMs fine-tuned on clinical data instead of general-purpose models appeared to reduce the risk of memorization. However, as memorization rates were not zero, organizations would be wise to not only continuously monitor LLM performance but also continuously fine-tune models on fresh data, Franklin University researchers said. The paper also recommended human-in-the-loop output validation and thorough data anonymization, not just deidentification.  

Brian Eastwood is a Boston-based writer with more than 10 years of experience covering healthcare IT and healthcare delivery. He also writes about enterprise IT, consumer technology, and corporate leadership.