Reproduce “SSD Offloading for LLM Mixture-of-Experts Weights Considered Harmful in Energy Efficiency” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kwanhee Kyung, Sungmin Yun, Jung Ho Ahn
Research from Seoul National University quantitatively demonstrates that offloading Mixture-of-Experts (MoE) Large Language Model weights to Solid State Drives (SSDs) increases per-token energy consumption by up to 12.5x compared to HBM, even with effective latency hiding. The analysis suggests a 10x improvement in Flash memory energy efficiency is needed for SSD offloading to become energetically competitive for large MoE models.
Key insights
- Offloading MoE weights to current Flash SSDs incurs an overwhelmingly dominant energy penalty per token, primarily due to the intrinsically higher read energy of Flash memory compared to HBM.
- Techniques designed to hide data transfer latency, such as prefetching, do not translate into energy savings, revealing a fundamental trade-off between latency and energy efficiency.
- Future improvements in Flash memory energy efficiency, specifically a tenfold reduction in read energy, are crucial to make SSD offloading energetically competitive for very large MoE models, especially when leveraging sparsity at small batch sizes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.