Reproduce “STAlloc: Enhancing Memory Efficiency in Large-Scale Model Training with Spatio-Temporal Planning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zixiao Huang, Junhao Hu, Hao Lin, Chunyang Zhu, Yueran Tang, Quanlu Zhang, Zhen Guo, Zhenhua Li and 4 more

STAlloc introduces a hybrid memory allocation paradigm that combines offline spatio-temporal planning with online allocation to reduce GPU memory fragmentation in large-scale deep learning model training. It achieves an average memory efficiency exceeding 95% and up to 56.3GB in GPU memory savings with negligible impact on training throughput, enabling training configurations previously limited by out-of-memory errors.

Key insights

  • GPU memory requests in large-scale deep learning exhibit strong spatio-temporal regularities, including repetitive tensor sizes (spatial) and predictable tensor lifespans (temporal), which can be leveraged for proactive planning.
  • A hybrid approach combining offline, ahead-of-time planning with online, runtime allocation can overcome the limitations of purely online allocators and the intractability of optimal offline planning.
  • By categorizing and planning allocations based on their fixed vs. dynamic nature and their computational phase lifespans, significant memory fragmentation can be mitigated while accommodating runtime variability.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.