Reproduce “Spectral-factorized Positive-definite Curvature Learning for NN Training” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wu Lin, Felix Dangel, Runa Eschenhagen, Juhan Bae, Richard E. Turner, Roger B. Grosse

The authors introduce a spectral method for maintaining positive-definite curvature preconditioners and computing fractional roots without repeatedly decomposing the full matrix. It learns an orthogonal basis and positive eigenvalues, with updates designed to preserve their constraints, and extends to Kronecker-structured preconditioners. In vision-transformer experiments, it matches Muon’s performance in BFP16, while SOAP takes longer; tests cover only roots p=1 and p=2.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.