Reproduce “Explaining the Roles of Experts in Mixture-of-Experts LLMs for Personalization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sekh Mainul Islam, Pepa Atanasova, Isabelle Augenstein

The authors analyze how personalization is represented in sparse mixture-of-experts language models. They tune routers and selected experts, then use a crosscoder to identify preference-related features and associate them with routed experts. In tests, targeted latent and expert interventions shift next-token distributions toward personalized models, beyond controls. The evidence covers synthetic group preferences and three MoE families.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.