Reproduce “XTRUST: On the Multilingual Trustworthiness of Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yahan Li, Yi Wang, Yi Chang, Yuan Wu

XTRUST introduces the first comprehensive multilingual benchmark to evaluate Large Language Model trustworthiness across 10 diverse categories and 10 languages. Its empirical evaluation of five commercial LLMs reveals significant performance disparities in trustworthiness, particularly in low-resource languages like Arabic, Russian, and Hindi, even for advanced models like GPT-4.

Key insights

  • No current Large Language Model (LLM) achieves uniformly reliable trustworthiness across all dimensions and languages evaluated by XTRUST, even top models like GPT-4.
  • Significant performance gaps exist, with LLMs consistently struggling in trustworthiness tasks for low-resource languages such as Arabic, Russian, and Hindi.
  • Current LLM development and alignment efforts appear to be biased towards high-resource languages, suggesting a need for more inclusive and targeted trustworthiness optimization for global deployment.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.