Reproduce “Meta-Learning Preferences for Multilingual LLM Alignment” on OpenResearch

About this paper

Jiaying Lin, Seongho Son, Nam Phuong Tran, Long Tran-thanh, Ilija Bogunovic, Debmalya Mandal

Researchers at the University of Warwick and collaborators developed a meta-learning framework, MAML-RLHF and MAML-DPO, to align multilingual Large Language Models (LLMs) with human preferences more efficiently. This approach, which frames language alignment as a multi-task adaptation problem, achieved win-rate improvements of up to 28% for low-resource languages with only 100 target-language preference samples, thereby enhancing global accessibility of advanced AI.

Key insights

• Multilingual preference alignment can be effectively formulated as a meta-learning problem, allowing efficient knowledge transfer from high-resource languages to improve sample efficiency for low-resource ones.
• The MAML framework provides a robust transferable initialization that consistently accelerates adaptation to new target languages, demonstrating effectiveness across varying linguistic distances and meta-training language compositions.
• Meta-learning LLM policies or reward models directly on Bradley-Terry losses offers provable convergence and significantly enhances alignment with minimal target-language data.

Summary by alphaXiv.