Reproduce “MixFP4: Enhancing NVFP4 with Adaptive FP4/INT4 Block Representations” on OpenResearch

About this paper

Jiaxiang Zou, Yonghao Chen, Ruilong Wu, Xinyu Chen

MixFP4, developed by researchers at The Hong Kong University of Science and Technology (Guangzhou), extends NVFP4 by adaptively selecting between FP4 E2M1 and E1M2 micro-formats per tensor block. This approach demonstrably enhances 4-bit quantized LLM accuracy, yielding improved perplexity and downstream task performance, with minimal hardware impact (3.1% area, 1.5% power overhead on Tensor Core datapath).

Key insights

• LLM tensor distributions exhibit significant spatial and intra-block heterogeneity, meaning a single, fixed low-bit micro-format is suboptimal for robust 4-bit quantization.
• A small set of two carefully chosen FP4 micro-formats (E2M1 and E1M2) is sufficient to capture most of the accuracy benefits from format adaptivity, avoiding unnecessary complexity.
• Hardware-efficient adaptive quantization is achievable by leveraging existing bits for metadata encoding and unifying internal computation paths, resulting in negligible overhead while maintaining compatibility with current Tensor Core designs.

Summary by alphaXiv.