Reproduce “SAE Interprets MoE: Sparse AutoEncoders Reveal That MoE Experts Exhibit Human-Interpretable Fine-Grained Specialization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Anyuan Zhuo, Shaohua Zhang, Yuan Lin, Pinyan Lu
The authors trace which experts drive learned features in mixture-of-experts model outputs by decomposing sparse autoencoder feature contributions across router-weighted experts. In Qwen3-30B-A3B, contributions are more concentrated than router weights across three layers, and interpreted dominant sets distinguish roles such as Java and front-end development. These results describe one model’s SAE-derived profiles and interpretations, not experimentally verified causal functions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.