Reproduce “Explaining the Roles of Experts in Mixture-of-Experts LLMs for Personalization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sekh Mainul Islam, Pepa Atanasova, Isabelle Augenstein
The authors analyze how personalization is represented in sparse mixture-of-experts language models. They tune routers and selected experts, then use a crosscoder to identify preference-related features and associate them with routed experts. In tests, targeted latent and expert interventions shift next-token distributions toward personalized models, beyond controls. The evidence covers synthetic group preferences and three MoE families.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.