Reproduce “When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tanguy Dieudonné, Giulia Lanzillotta, Enis Simsar, Louis Barinka, Thomas Hofmann
LITELORA, developed by researchers at ETH Zurich, introduces a gating framework for continual fine-tuning that dynamically determines whether to reuse existing low-rank adapters or recruit new ones for each incoming task. This method reduces parameter growth by up to 70% while maintaining or improving average accuracy on continual learning benchmarks like ImageNet-A and ImageNet-R.
Key insights
- Task-specific LoRA adapters exhibit significant low-rank redundancy, implying that the subspaces spanned by adapters trained on different tasks often overlap, and prior adapters can represent subsequent tasks.
- The low-rank subspace needed to represent a stream of N tasks can be spanned by significantly fewer than N adapters, suggesting that excessive capacity can sometimes increase the risk of forgetting.
- Dynamically adjusting model capacity based on the semantic novelty of a task, rather than a fixed 'one-adapter-per-task' rule, is crucial for efficient and effective continual learning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.