Reproduce “HSAP: A Hierarchical Sequence-aware Parallelism for Hybrid-Context Generative Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Songxin Zhang, Zejian Xie, Zhuoyang Song, Cong lin, Junyu Lu, Jiaxing Zhang, Bingyi Jing
HSAP, a hierarchical sequence-aware parallelism framework, enables efficient training of large generative models on ultra-long and hybrid-context packed sequences. It leverages a novel Sequence-Aware Parallelism (SAP) algorithm with JIT compilation to correctly compute causal attention on packed data, achieving increased throughput and scalability, allowing for training with up to 512K token sequences without Out Of Memory errors.
Key insights
- JIT compilation within SAP provides dynamic adaptation to diverse attention patterns in hybrid-context packed sequences, enabling correct causal attention without complex hardcoded logic.
- Combining InterSP and IntraSP hierarchically allows for superior scalability and efficiency by addressing their individual limitations and maximizing parallelism for ultra-long sequences.
- Fine-grained control over P2P communications, operation fusion, and optimized network topology utilization significantly reduce communication overhead and improve computational efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.