Reproduce “Stiefel-AdamW: Geometry-Aware AdamW for Linear Factorization Blocks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Emanuele Zangrando, Marco Sutti, Francesco Tudisco
Stiefel-AdamW constrains one factor to have orthonormal rows, leaving the other unconstrained. It retains AdamW-style adaptive updates, projecting and retracting the constrained factor’s step. This limits unbounded factor rescaling but retains orthogonal invariances. Experiments report lower GPT-2 test loss at equal peak memory and higher GLUE average in Mistral 7B LoRA fine-tuning than AdamW. These results are limited to reported settings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.