Reproduce “DFRot: Achieving Outlier-Free and Massive Activation-Free for Rotated LLMs with Refined Rotation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jingyang Xiang, Sai Qian Zhang

DFRot refines a rotation in a 4-bit language-model quantization pipeline by weighting rare massive-activation tokens more heavily during calibration while retaining the need to represent ordinary tokens. This targets a long-tail source of quantization error without proving such activations are eliminated. On LLaMA3-70B in W4A4KV16, WikiText-2 test perplexity is 4.97 versus QuaRot’s 5.92, but not best overall.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.