Reproduce “Scale-QLoRA: Code-Invariant Adapter Merging for Native 4-bit Microscaling LLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tung-Ling Li, Jiale Huang, Lee-Chi Wang, Janaki Ram Gotei

Scale-QLoRA adapts a task model for a standalone native 4-bit checkpoint by learning low-rank block-scale corrections while freezing weight codes. Quantization-aware scale training rounds scales to the target grid during training, enabling a bit-exact merge within that format and grid. On Llama-3.1-8B NVFP4 Banking77, it scores 93.7% after merge, versus 55.6% for naive merging; grid conversion is not covered.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.