Reproduce “Split-KV: Efficient Context Parallelism for Training Diffusion Drafters” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Geng Zhang, Fangzhao Zhang, Cong Xie, Tiantian Fan, Yang You
Split-KV trains diffusion drafters on long contexts by keeping key/value shards local, communicating query-sized tensors per attention layer, and assigning context chunks by attention cost. In controlled studies, Qwen3.5-35B-A3B’s mean acceptance length varied 0.82% across 4k to 64k maximum context limits at fixed query length. At context-parallel size 64, it achieved 3.67× faster steps than LV-XAttn for the specified workload, not inference speed or answer accuracy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.