Reproduce “Modular Norm RandOpt: Population-Efficient Ensembling through Architecture-Aware Perturbations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kirato Yoshihara, Hiroaki Hamade
Modular Norm RandOpt is a gradient-free method for adapting pretrained language models. It replaces RandOpt’s global perturbation scale with role-matched norms and calibrated module scales, while retaining candidate scoring, top-K selection, and plurality voting. On GSM8K, Qwen2.5-1.5B-Instruct with K=25 used N=25 candidates and exceeded RandOpt’s N=300 mean held-out accuracy by 3.11 percentage points; gains vary across tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.