Reproduce “TileQ: Efficient Low-Rank Quantization of Mixture-of-Experts with 2D Tiling” on OpenResearch

About this paper

Hongyaoxing Gu, Xinzhe Chen, Lijuan Hu, Fangfang Liu

TileQ introduces a post-training quantization method for Mixture-of-Experts (MoE) models, leveraging 2D-tiling structured low-rank decomposition and a fused sparse inference scheme. This approach substantially reduces memory footprint by up to 8x for low-rank factors and accelerates inference by up to 20x during the decode phase, while maintaining accuracy at 2-bit and 3-bit precision without requiring fine-tuning.

Key insights

• Jointly sharing low-rank factors across both input and output dimensions of MoE experts through a 2D tiling strategy effectively reduces the memory overhead of the low-rank components by an order of magnitude compared to 1D or per-expert methods.
• Fusing irregular sparse computations into structured, batched operations significantly improves inference latency in MoE models, particularly during prefill (less than 5% overhead) and decode (up to 20x speedup), by optimizing hardware utilization.
• High quantization accuracy for MoE models at aggressive low bit-widths (e.g., 2-bit, 3-bit) can be achieved without fine-tuning by combining a robust 2D-tiling low-rank decomposition with effective residual quantization strategies.

Summary by alphaXiv.