Reproduce “Task Arithmetic with LoRA for Continual Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Rajas Chitale, Ankit Vaidya, Aditya Kane, Archana Ghotkar

Researchers at Pune Institute of Computer Technology developed a continual learning method for Vision Transformers that largely bypasses catastrophic forgetting while drastically reducing computational cost. Their approach integrates Low-Rank Adaptation (LoRA) for efficient task-wise training, task arithmetic to combine learned knowledge without interference, and a final fine-tuning step on a small memory set, achieving performance comparable to offline training with significantly fewer computational resources.

Key insights

  • Combining LoRA for efficient task-wise training with task arithmetic for knowledge aggregation can effectively bypass catastrophic forgetting in continual learning.
  • A final fine-tuning step on a small, representative memory set can consolidate learned knowledge from task arithmetic, bringing performance closer to offline learning.
  • The utility of regularization techniques like KL-Divergence loss can be dataset-dependent, offering benefits in high-variance multi-class scenarios but potentially over-regularizing simpler ones.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.