Reproduce “SEA-SafeguardBench: Evaluating AI Safety in SEA Languages and Cultures” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Panuthep Tasawong, Jian Gang Ngui, Alham Fikri Aji, Trevor Cohn, Peerat Limkonchotiwat
A new benchmark, SEA-SafeguardBench, evaluates AI safety in Southeast Asian languages and cultures, revealing that current models struggle with cross-lingual generalization and culturally nuanced harm detection. The evaluation of 20 LLMs and safeguard models showed performance drops of 5.4 to 6.1 AUPRC points on SEA languages and substantial degradation (up to 36.4 AUPRC points) on culturally specific harm scenarios.
Key insights
- Current safeguard models exhibit limited cross-lingual generalization, consistently underperforming in typologically diverse SEA languages compared to English.
- Safeguards critically lack the ability to understand and align with nuanced cultural sensitivities and region-specific taboos, leading to substantial performance degradation on such content.
- Effective cultural alignment in LLMs requires foundational region-specific pre-training knowledge, as cultural instructions alone offer only marginal gains for models without prior cultural exposure.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.