Reproduce “Unlocking the Potential of Large Language Models for Clinical Text Anonymization: A Comparative Study” on OpenResearch

About this paper

David Pissarra, Isabel Curioso, João Alveira, Duarte Pereira, Bruno Ribeiro, Tomás Souper, Vasco Gomes, André V. Carreiro and 1 more

Researchers from Fraunhofer AICOS demonstrated that fine-tuned Large Language Models can effectively anonymize clinical text, balancing robust privacy protection with high retention of clinical information. The study introduces six novel evaluation metrics tailored for generative anonymization methods, showing that fine-tuned generative LLMs achieve competitive privacy scores and the highest data utility compared to traditional methods.

Key insights

• Fine-tuning is crucial for generative LLMs, with fine-tuned models significantly outperforming their zero-shot counterparts in both privacy and utility, even for smaller models.
• Fine-tuned generative LLMs achieved a strong balance, offering competitive anonymization sensitivity and the highest clinical information retention among all tested methods.
• ClinicalBERT, an encoder-only LLM, led in anonymization sensitivity metrics, while KNEO, despite high string-matching recall, severely compromised clinical information utility.

Summary by alphaXiv.