Reproduce “FireQ: Fast INT4-FP8 Kernel and RoPE-aware Quantization for LLM Inference Acceleration” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Daehyeon Baek, Jieun Choi, Jimyoung Son, Kyungmin Bin, Seungbeom Choi, Kihyo Moon, Minsung Jang, Hyojung Lee
FireQ pairs INT4 weights with FP8 activations in kernels designed for H100 GPUs to improve inference throughput. Its linear-layer kernel converts weights through an in-register lookup table before FP8 tensor-core multiplication, while calibration and smoothing address quantization errors. On Llama2-7B at batch size 16, it achieves 1.68× QServe’s FFN-layer throughput, a layer-level result measured on one H100, not an end-to-end serving gain.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.