Reproduce “Multi LoRA Meets Vision: Merging multiple adapters to create a multi task model” on OpenResearch

About this paper

Ege Kesim, Selahattin Serdar Helli

Researchers from Huawei R&D Turkiye demonstrate that Low-Rank Adaptation (LoRA) adapters trained on different computer vision tasks can be merged to form a unified multi-task model without requiring explicit multi-task training. Experiments show that while some performance degradation occurs, merged models often outperform a head-only fine-tuning baseline, and task compatibility is higher for visually dissimilar tasks.

Key insights

• Merging LoRA adapters generally leads to some performance degradation compared to individual task-specific adapters, but often still surpasses a baseline of fine-tuning only the task-specific heads.
• The compatibility of merged adapters depends on task similarity; visually dissimilar tasks tend to merge with less performance interference, suggesting more orthogonal parameter modifications.
• Face-related tasks, particularly facial landmark detection, show higher interference when merged, likely due to conflicting feature representations or high-dimensional output requirements.

Summary by alphaXiv.