Reproduce “Fast and Memory Efficient Offload Training Framework with Hybrid XPU Computation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhiyi Yao, Zuning Liang, Yuedong Xu, Jin Zhao, Jessie Hui Wang, Tong Li
MemFerry schedules layer-by-layer computation between direct host access to CPU memory and GPU-staged parameters, using per-layer profiles to overlap work with data movement. It also supports gradient writes to CPU memory. In a 4.3-billion-parameter model at eight GPUs, it reports 19.1% higher training speed than native ZeRO-Offload, a system-level result that does not isolate DHA’s effect or generalize automatically.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.