Reproduce “Low-Rank Friction for Memory-Efficient Transformer Pretraining” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rajit Rajpal, Benedict Leimkuhler
R-iKFAD reduces the memory required for iKFAD’s adaptive friction by replacing its parameter-wise tensor with a rank-1 estimate built from row and column statistics of past squared momentum, while retaining the full momentum buffer. Across four tasks, it matches or improves on iKFAD0 test loss and uses about half the auxiliary optimizer state; this does not establish superiority over Adam.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.