Reproduce “Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Kaiyue Wen, Xingyu Dang, Kaifeng Lyu, Tengyu Ma, Percy Liang

The paper introduces Muon Hyperball, which constrains matrix weight norms and normalizes optimizer updates. It projects attention and MLP matrices back to their initial Frobenius norms, while other parameters use Adam. In 1.2B-parameter Qwen3-style pretraining, it reports 20–30% token-equivalent speedup versus about 10% for Muon with weight decay, based on fitted AdamW scaling laws, not wall-clock time.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.