Reproduce “Task-Aware Low-Rank Adaptation of Segment Anything Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xuehao Wang, Feiyang Ye, Yu Zhang
This work introduces Task-Aware Low-Rank Adaptation (TA-LoRA), a parameter-efficient fine-tuning approach that enables the Segment Anything Model (SAM) to simultaneously perform multiple diverse computer vision tasks. The method leverages tensor decomposition to efficiently learn task-shared and task-specific information, outperforming traditional multi-task learning methods and naive PEFT applications with a +23.93% improvement over the HPS baseline on NYUv2 while maintaining parameter efficiency.
Key insights
- SAM's pre-trained features can be effectively transferred to a broader range of dense prediction tasks beyond its original segmentation scope, enabling its use as a versatile multi-task vision foundation model.
- Tensor decomposition, specifically Tucker decomposition, effectively disentangles and learns both task-shared and task-specific information within a compact parameter space for multi-task parameter-efficient fine-tuning (PEFT).
- Optimal multi-task performance requires capturing both task-shared and task-specific knowledge, as methods that only share or only specialize perform suboptimally.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.