Reproduce “NyoomFloat12: Accelerating LLM Inference via Lossless 12-Bit Weight Compression” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sylvie Liberman, Xinyu Fang, Tianyi Zhang, Tri Dao, Daniel Y. Fu
The authors introduce NF12, a lossless format that stores BF16 weights in 12 bits and reconstructs them in a GPU-friendly, fixed-length pattern. It exploits concentrated exponent values and handles rare exceptions separately. On a shared-prefix Qwen3-32B rollout workload, NF12 reaches a peak of 787 output tokens per second versus BF16’s 310, a schedule-dependent 2.54× ratio; gains are not universal.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.