Reproduce “Scale-QLoRA: Code-Invariant Adapter Merging for Native 4-bit Microscaling LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tung-Ling Li, Jiale Huang, Lee-Chi Wang, Janaki Ram Gotei
Scale-QLoRA adapts a task model for a standalone native 4-bit checkpoint by learning low-rank block-scale corrections while freezing weight codes. Quantization-aware scale training rounds scales to the target grid during training, enabling a bit-exact merge within that format and grid. On Llama-3.1-8B NVFP4 Banking77, it scores 93.7% after merge, versus 55.6% for naive merging; grid conversion is not covered.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.