Reproduce “Unraveling Babel: Exploring Multilingual Activation Patterns of LLMs and Their Applications” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Weize Liu, Yinlong Xu, Hongxia Xu, Jintai Chen, Xuming Hu, Jian Wu
Researchers at Zhejiang University, UIUC, and HKUST explored the internal multilingual processing of Large Language Models by converting them into fine-grained Mixture-of-Experts architectures. They identified distinct language-specific and shared activation patterns across layers and used these findings to reduce LLM computational costs by up to 18% with preserved or improved performance.
Key insights
- LLM internal activation patterns for different languages are closely related to linguistic properties like language families, genera, and shared alphabet systems.
- Multilingual shared experts, processing language-independent knowledge, are predominantly concentrated in the middle layers of LLMs.
- Instruction tuning primarily induces changes in expert activation frequencies within the last layer of the model.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.