Reproduce “Decode Token Parallelism: Memory-Efficient Distributed Attention Parallelism for Scalable LLM Decoding” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Susav Shrestha, Brad Settlemyer, Narasimha Reddy

The authors present TKNP, a decode-serving method that places balanced blocks of one model copy across nodes while assigning each node requests and their KV caches. Nodes compute attention locally and exchange current-token activations for dense-layer work. For Llama 3.1 405B on eight H100 nodes, single-copy TKNP reaches up to 1.53× whole-workload completion throughput over DCP, but can lose at low concurrency.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.