Reproduce “Making LoRA Identifiable: Orthogonal Alignment for Continual Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Pratik Rakesh Singh, Mohammadi Zaki, Akash Saha, Aneesh Mukkamala, Pankaj Wasnik
SOAR consolidates sequential task updates into one low-rank adapter for a frozen model, keeping adapter storage fixed as task count grows. It constrains one factor to have orthonormal rows, aligns the other factor across tasks, and selectively retains compatible past directions. On Qwen3-8B, its SuperNI average is 51.59 versus 47.84 for SLAO; it cannot recover knowledge fully overwritten during adaptation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.