Reproduce “Orthogonal Low-rank Adaptation in Lie Groups for Continual Learning of Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kefan Cao, Shuaicheng Wu
OLieRA addresses sequential continual learning with low-rank task updates, applied multiplicatively through an elementwise exponential and constrained against earlier updates. On the five-task Standard CL benchmark, T5-large reaches 79.6 average accuracy after the final task, versus 78.8 for N-LoRA and 80.0 for jointly trained MTL. The lead is modest and remains below MTL.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.