Reproduce “Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Junzhuo Li, Bo Wang, Xiuze Zhou, Xuming Hu

Dynamic Expert Specialization (DES-MoE) offers a framework for Mixture-of-Experts (MoE) models to adapt to multiple diverse domains without catastrophic forgetting. It achieved an average score of 51.6 across six specialized tasks in a mixed-domain setting, outperforming full-parameter fine-tuning by 5.0 points, while retaining general abilities identical to the vanilla pre-aligned LLM baseline.

Key insights

  • Dynamically adapting a lightweight router with a combined distillation and task adaptation loss, phased over training, effectively enables domain-specific routing without sacrificing pre-trained general knowledge.
  • Explicitly mapping expert-domain affinities and using gradient masking to isolate expert updates is crucial for preventing catastrophic forgetting and cross-domain interference in multi-domain MoE adaptation.
  • A progressive freezing schedule, which starts with broader parameter updates and gradually narrows to expert-only updates, provides a stable and parameter-efficient approach for specializing MoE models across multiple domains.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.