Reproduce “LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Inconsistencies” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Felix Friedrich, Simone Tedeschi, Patrick Schramowski, Manuel Brack, Roberto Navigli, Huu Nguyen, Bo Li, Kristian Kersting

M-ALERT extends the ALERT safety benchmark to five languages by translating 15,000 red-team prompts per language and scoring responses with a multilingual judge. It reports safety overall and across 32 risk categories, plus exact response matching across languages. In crime_tax, Llama-3-8B-Instruct scores 100.0% safe in English and 67.68% in Italian, but results are limited to these languages and automated judgments.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.