Reproduce “Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ziqi Yuan, Haoyang Zhang, Yirui Eric Zhou, Apoorve Mohan, I-Hsin Chung, Seetharami Seelam, Jian Huang
Teraio, developed by the University of Illinois Urbana-Champaign and IBM Research, is a system that intelligently offloads inactive LLM tensors to low-cost SSDs using GPUDirect Storage. This approach enables training of models like Llama3-70B that exceed GPU memory, achieving 1.47x better average performance than state-of-the-art solutions and reducing hardware costs by 5.88x for large models.
Key insights
- Only a small fraction (on average, 1.7%) of allocated GPU memory contains tensors actively used by GPU kernels at any given time, presenting opportunities for offloading.
- Many large tensors (>10MB) remain inactive for extended periods (>10,000 microseconds), providing ample time for offloading to slower storage without performance impact.
- Larger tensors tend to have longer inactive periods, making them ideal candidates for offloading to maximize memory savings while minimizing performance effects.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.