Reproduce “Janus-LoRA: A Balanced Low-Rank Adaptation for Continual Learning” on OpenResearch

About this paper

Cheng Chen, Pengpeng Zeng, Yuyu Guo, Lianli Gao, Hengtao Shen, Jingkuan Song

Janus-LoRA introduces a framework for continual learning that addresses catastrophic forgetting in LoRA-based models by resolving parameter-level misalignment and feature-space encroachment. It achieves new state-of-the-art performance across various benchmarks, including a 77.11% MAA on ImageNet-R (20 tasks), while demonstrating superior training efficiency.

Key insights

• Catastrophic forgetting in LoRA-based CL is fundamentally caused by both the internal misalignment of independent low-rank factor updates and the encroachment of new features into existing latent spaces.
• Orthogonality in parameter updates, if properly enforced (via GR), can significantly mitigate forgetting, while online estimation (OE) provides an efficient, memory-free way to define the historical knowledge subspace.
• Explicitly preserving separation in the feature space (via DML) is crucial for enhancing plasticity without exacerbating forgetting, as it prevents new concepts from ambiguously overlapping with old ones.

Summary by alphaXiv.