Reproduce “Shared Low-rank Basis Factorization for Data-free Mixture-of-Experts Compression” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianxiao Cao, Jiahe Shao, Yuning Qiu, Kyohei Atarashi, Hisashi Kashima, Qibin Zhao
SLBF compresses mixture-of-experts language models by representing each expert’s weights with an expert-specific adapter and mixtures of shared, factorized low-rank bases, while preserving experts and routing. It fits weights without calibration data or post-compression fine-tuning. On Qwen3-30B-A3B, it reduces total parameters by 24% and averages 84.3 across nine benchmarks, versus 84.9 uncompressed; this aggregate does not preserve every task score.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.