Reproduce “Mixture of Modular Experts: Distilling Knowledge from a Multilingual Teacher into Specialized Modular Language Models” on OpenResearch

About this paper

Mohammed Al-Maamari, Mehdi Ben Amor, Michael Granitzer

Researchers from the University of Passau integrated Knowledge Distillation with various Mixture of Experts architectures to develop specialized, modular language models from a multilingual teacher. This approach successfully created efficient student models for English, French, German, and Python while completely eliminating catastrophic forgetting across languages.

Key insights

• Modular Mixture of Experts (MoE) architectures completely eliminated catastrophic forgetting across languages, demonstrating superior knowledge retention compared to sequential training methods.
• The inclusion of a common expert in MoE systems demonstrably improved overall performance, reducing perplexity by 5.69 points for the MoE-CE architecture when enabled.
• Both fixed and adaptive alpha methods in Knowledge Distillation yielded similar performance, with combined loss functions marginally outperforming alternating losses.
• The use of a pre-trained, high-accuracy router is crucial for efficient and correct dispatching of multilingual inputs to specialized experts, optimizing resource allocation.

Summary by alphaXiv.