Reproduce “Guess-Verify-Refine: Data-Aware Top-K for Sparse-Attention Decoding on Blackwell via Temporal Correlation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Long Cheng, Ritchie Zhao, Timmy Liu, Mindy Li, Xianjie Qiao, Kefeng Duan, Yu-Jung Chen, Xiaoming Chen and 2 more
NVIDIA researchers developed Guess-Verify-Refine (GVR), an exact Top-K algorithm for sparse-attention decoders that exploits temporal correlation in Top-K index sets across consecutive decode steps. The method achieves an average 1.88x single-operator speedup and up to 7.52% end-to-end throughput improvement for DeepSeek-V3.2 on NVIDIA Blackwell GPUs without degrading model accuracy.
Key insights
- A strong temporal correlation exists between Top-K index sets in consecutive sparse-attention decode steps, attributable to the RoPE/YaRN structure of the indexer's attention mechanism.
- Data-aware algorithms tailored to exploit workload-specific properties, like temporal stability, can outperform general-purpose GPU primitives in critical LLM inference components.
- Efficient GPU kernel design benefits from a single-CTA approach with extensive shared memory usage and ballot-free operations to minimize global memory bandwidth and serialization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.