Reproduce “ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuchen Yang, Yaru Zhao, Pu Yang, Shaowei Wang, Zhi-Hua Zhou
ZipMoE presents an on-device Mixture-of-Experts (MoE) serving system designed for unified memory architecture (UMA) edge devices. It integrates lossless compression of expert parameters with a cache-affinity scheduling and parallel decompression engine, reducing inference latency by up to 72.77% and boosting throughput by up to 6.76x without altering model behavior.
Key insights
- MoE expert parameters, particularly BF16 exponent bits, exhibit significant statistical redundancy and high compressibility, allowing for lossless compression to reduce storage requirements.
- On UMA-based edge devices, SSD I/O is the primary bottleneck for MoE inference, but multi-threaded CPU decompression can be faster than I/O and effectively overlapped to hide latency.
- A hierarchical caching strategy combined with a scheduling mechanism that accounts for compression states and leverages CPU parallelism can transform I/O-bound MoE inference into a compute-centric workflow.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.