Reproduce “FOSCAR: Accelerating Decode Attention over 2- and 4-Bit KV Caches through Affine Folding” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Donglin Zhuang, Pragaash Ponnusamy, Xiaoxia Wu, Shuaiwen Leon Song, Zhongzhu Zhou

FOSCAR accelerates low-bit key–value cache attention by folding group-shared scales and offsets into the query-key and probability-value calculations, avoiding per-value reconstruction. The rewrites preserve the product of a given affine-quantized cache in exact arithmetic, not its quantization accuracy. On B200, INT2 achieves a 2.67× geometric-mean kernel-latency speedup over FlashInfer BF16 across tested batch and context settings.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.