Reproduce “Multi-GPU MoE Inference with Adaptive Expert Offloading” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
William X. Zheng, Zhongjie Chen, Shuo Yang, Ziming Mao, Xiaoze Fan, Yi Pan, Kunjun Li, Hanfei Yu and 4 more
AdaOffload coordinates CPU and GPUs to serve mixture-of-experts models under limited GPU memory. It profiles hardware and workload, then adapts whether cache-missed experts are fetched to GPUs or computed on CPUs. On GCP L4 GPUs serving eight concurrent AIME tasks, decode throughput rises from 28.5 tok/s with one GPU to 90.8 with eight; this result is specific to the tested setup.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.