Reproduce “SlimPack: Fine-Grained Asymmetric Packing for Balanced and Efficient Variable-Length LLM Training” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuliang Liu, Guohao Wu, Shenglong Zhang, Wei Zhang, Qianchao Zhu, Zhouyang Li, Chenyu Wang

SlimPack is a distributed LLM training framework that achieves up to 2.8x throughput improvement by implementing fine-grained, slice-level data packing and asymmetric partitioning. It effectively balances computational workloads across both forward and backward passes in hybrid parallel systems, leading to more efficient hardware utilization with variable-length inputs.

Key insights

  • Decomposing samples into flexible, fine-grained slices (MicroPacks) is essential for achieving precise workload balancing and eliminating straggler effects in distributed LLM training.
  • Explicitly addressing the asymmetric computational costs between forward and backward passes through dynamic re-partitioning is crucial for maintaining continuous pipeline flow and overall efficiency.
  • A holistic co-design of fine-grained packing, asymmetric scheduling, and high-fidelity simulation can simultaneously resolve workload imbalances across all parallelism dimensions (DP, PP, CP) without introducing significant memory or communication overheads.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.