Reproduce “Factored One-sided Adam-Moment for Practical and Scalable SOAP” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jaemyung Yu, Byeongho Heo, Sangdoo Yun, Dongyoon Han
FOAM reduces SOAP’s cost by retaining input-side preconditioning and summarizing rotated second moments with row and column means. In GPT-2 large (760M) FineWeb pre-training, FOAM’s final validation loss was 2.796 ± 0.005 nats, close to SOAP’s 2.791 ± 0.005. It reached AdamW’s final-loss target in 10.53 hours versus SOAP’s 22.66, using 11.87 versus 23.73 GiB of matrix-parameter state; evidence is limited to GPT-2-scale models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.