Reproduce “BitNest: Bit-Nested Speculative Decoding for Memory-Efficient LLM Inference Acceleration” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chence Yang, Ningxi Cheng, Arash Akbari, Qitao Tan, Qingchan Zhu, Ci Zhang, Changdi Yang, Yanzhi Wang and 4 more
BitNest uses one nested weight representation for speculative decoding, with a low-precision draft and a refined, higher-precision target. It fixes a 4-bit weight base, stores a 4-bit residual refinement separately, and reads the base alone for drafting or both planes for verification. On LLaMA-2-7B, W4A8-to-W8A8 decoding is 1.59× faster on GSM8K than 16-bit autoregressive decoding; comparisons use native method configurations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.