Reproduce “LASP-2: Rethinking Sequence Parallelism for Linear Attention and Its Hybrid” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Weigao Sun, Disen Lan, Yiran Zhong, Xiaoye Qu, Yu Cheng
LASP-2 reorganizes linear-attention sequence parallelism by computing chunks locally and gathering compact memory states in one collective per forward pass, with a gradient collective in the backward pass. In a Linear-Llama3 setup with batch size 1 and 64 A100 GPUs, it reports 36.6% higher throughput than Ring Attention at 2048K tokens; this establishes efficiency only for the tested setup.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.