Reproduce “CLINIC: Evaluating Multilingual Trustworthiness in Language Models for Healthcare” on OpenResearch

About this paper

Akash Ghosh, Srivarshinee Sridhar, Raghav Kaushik Ravi, Muhsin Muhsin, Sriparna Saha, Chirag Agarwal

CLINIC introduces the first comprehensive multilingual benchmark to evaluate the trustworthiness of language models in healthcare across 15 languages and five critical dimensions, revealing that proprietary models generally outperform open-weight and medical-specific models, with consistent performance degradation observed in mid- and low-resource languages.

Key insights

• Current language models, including specialized medical LLMs, exhibit significant trustworthiness deficiencies across multiple dimensions, such as high hallucination rates, biases in fairness, and privacy vulnerabilities, making them unreliable for critical multilingual healthcare applications.
• Performance consistently degrades from high-resource to mid- and low-resource languages across nearly all trustworthiness tasks, underscoring a critical linguistic bias that impacts global healthcare AI equity.
• Proprietary models generally demonstrate superior trustworthiness compared to open-weight and medical-specific models, but even top-performing models show weaknesses in specific areas, such as privacy leakage.

Summary by alphaXiv.