Reproduce “Rimuru: Unified Memory Management for Modern LLM Serving via Elastic Expert Offloading” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhiyuan Fang, Dingyang Li, Jianfeng Zhong, Bin Wang, Weixi Zhu, Cunshu Ni, Zicong Hong, Wuhui Chen and 1 more
Rimuru shares GPU pages between cached expert weights and growing attention states, shifting capacity as request demand changes. Its allocator and page-aware kernel support this sharing. In a Qwen1.5-MoE-14B test on an RTX 5090 at 10 requests per second, concurrency rose from 74 to 90 and forward steps fell from about 1,300 to 1,000 versus fixed-cache offloading; these steps are not elapsed time.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.