Reproduce “Ori: Drift-Aware Streaming $k$-PCA for Very Low-Rank LLM Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinyu Luo, Jincheng Zhou, Mengbo Wang, Zilin Shen, Nadia Atallah Lanman, Petros Drineas, Bruno Ribeiro, Brian Bullins
Ori tracks changing gradient subspaces for low-rank language-model training, aiming to reduce optimizer-state memory without letting its projection basis become stale. It warm-starts basis updates from the previous basis and refines them with streaming Oja++ updates. On 1B LLaMA pretraining on C4 at rank 128, Ori reaches validation perplexity 21.61, versus OSD’s 25.43; it remains behind full-rank Adam at 19.92.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.