Reproduce “Zeta: Dual Whitening for Matrix Optimization via Coordinate-Adaptive Preconditioning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kaiwen Chen, Shuhai Zhang, Zimo Liu, Linxiao Li, Ying Sun, Yuchen Li, Yifan Zhang, Bo Han and 2 more
Zeta combines coordinate-wise normalization with matrix-level orthogonalization for neural-network training. It divides momentum entries by their running second moments before applying Newton–Schulz iterations to matrix-shaped weights. On Qwen3-1.7B, Zeta reaches the same training loss 1.67× faster by iteration count and 1.64× faster by elapsed time than AdamW, and reports the best average accuracy across 12 benchmarks in tested settings, not every task.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.