Reproduce “UCCL-Zip: Lossless Compression Supercharged GPU Communication” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shuang Ma, Chon Lam Lao, Zhiying Xu, Zhuang Wang, Ziming Mao, Delong Meng, Jia Zhen, Jun Wu and 3 more

UCCL-Zip improves GPU communication by integrating lossless compression into peer-to-peer transfers and NCCL collectives. Uzip-P2P sends a tensor portion as soon as it is ready while compressing the remainder; Uzip-NCCL fuses compression into the communication kernel. On GLM4-9B’s 214 MB gate_up_proj tensor, throughput rose up to 47.5% over UCCL-P2P; this is not whole-job speedup.

Key insights

  • Overlapping initial data transmission with subsequent compression steps effectively mitigates the non-linear latency scaling of GPU compression for point-to-point communication.
  • Adapting compression algorithms to collective communication frameworks, specifically by utilizing localized frequency tables and directly integrating into existing data paths, reduces memory traffic and kernel launch overheads.
  • While beneficial for P2P and certain collective operations, the overhead of current lossless compression can outweigh benefits for fine-grained, multi-step collectives (like ring All-Reduce) or high-bandwidth intra-node links (NVLink), indicating a need for further co-design.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.