Reproduce “TileQuant: Tile as the Precision-Allocation Unit for LLM Mixed-Precision Post-Training Quantization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hanzhi Zhang, Qiao Zhang, Qinglei Cao, Heng Fan, Yan Huang, Kewei Sha, Yunhe Feng
TileQuant uses an offline calibration map to retain important two-dimensional weight tiles in 16-bit precision and quantize the rest to 8-bit, enabling mixed-precision inference with regular tile computation. On Llama-3.2-3B-Instruct, one batch-1 request configuration reaches 154.98 tokens per second, 2.66 times BF16; this is complete-request throughput under the tested conditions, not a universal speedup.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.