Reproduce “MuPACE: Learning to Pace Muon for LoRA Fine-Tuning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jan Kobiolka, Christian Frey, Josif Grabocka
MuPACE adapts learning rates during LoRA fine-tuning by adjusting step size while keeping Muon’s update direction fixed. A recurrent controller observes training and validation signals and modifies a bounded analytic schedule. Across 64 configurations, it beats a shared cosine schedule on 50 tasks, with a median relative test-loss reduction of 3.93%; against tuned schedules and on larger models, results are competitive but not uniformly better.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.