Reproduce “BitDecoding: Unlocking Tensor Cores for Long-Context LLMs with Low-Bit KV Cache” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dayou Du, Shijie Cao, Jianyi Cheng, Luo Mai, Ting Cao, Mao Yang
BitDecoding, developed by researchers from the University of Edinburgh, Microsoft Research, and Tsinghua University, presents an inference system that orchestrates CUDA and Tensor Cores for efficient low-bit Key-Value (KV) cache decoding. This system delivers up to 8.6x speedup over FP16 baselines and over 4x higher throughput for long-context Large Language Models, while incurring minimal accuracy degradation.
Key insights
- It is possible to generate Tensor Core-compatible layouts for dynamic low-bit data by fusing computation, quantization, and packing within Tensor Core fragments, eliminating costly explicit remapping.
- Optimized warp-level parallelism, where multiple warps concurrently dequantize different data chunks, effectively hides dequantization latency and ensures continuous Tensor Core utilization.
- Fine-grained asynchronous pipelines can overlap CUDA core dequantization with Tensor Core matrix multiplication, maximizing the utilization of both core types and reducing overall kernel execution time.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.