Reproduce “Cross-Lingual Alignment for Decoder-Only Models using MoE Routers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Lucas Bandarkar, Clark Peng, Ahmed Haj Ahmed, Aditi Khandelwal, Nanyun Peng

The method aligns translated sentences in decoder-only mixture-of-experts models by comparing sequence-level expert-use profiles rather than token states. It averages router weights across each sequence and adds a routing loss to next-token training. Across four post-trained models and seven languages, it improved average benchmark scores in 13 of 14 pairs and tied once, in this limited continued-pretraining setting.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.