Reproduce “How Does Orthogonalization Adapt to the Neural Network Hessian Structure? A Gradient Self Outer-Product Analysis at Initialization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shenyang Deng, Garvit Goyal, Tianyu Pang, Shuhua Yu, Yaoqing Yang

The paper explains how Muon’s row-mixing update can align with neural network curvature: at Gaussian initialization, gradient rows become weakly correlated as dimensions grow. It analyzes four model families and measures increasing diagonal dominance across four attention projections in three language models as width grows. These are initialization results, not evidence about training dynamics or optimization speed.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.