Reproduce “CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs” on OpenResearch

About this paper

Sijia Chen, Xiaomin Li, Mengxue Zhang, Eric Hanchen Jiang, Qingcheng Zeng, Chen-Hsiang Yu

CARES, a comprehensive benchmark for evaluating medical large language model safety, was introduced, revealing vulnerabilities of existing models to adversarial prompts and demonstrating that a jailbreak-aware classifier can improve safety alignment. The benchmark integrates eight medical safety principles, four graded harm levels, and three jailbreak strategies to provide a nuanced assessment.

Key insights

• State-of-the-art large language models (LLMs) consistently demonstrate significant vulnerability to adversarial rephrasing ("jailbreak" attacks), leading to degraded safety performance.
• Jailbreak attempts not only make harmful prompts appear less harmful but can also cause safe prompts to be misidentified as risky, particularly challenging for models across harmless (Level 0) and highly harmful (Level 3) categories.
• The integration of a jailbreak detection classifier and reminder prompts effectively enhances LLM safety alignment, yielding substantial performance boosts for models with weaker inherent safety mechanisms.

Summary by alphaXiv.