Reproduce “Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yunhao Gou, Zhili Liu, Kai Chen, Lanqing Hong, Hang Xu, Aoxue Li, Dit-Yan Yeung, James T. Kwok and 1 more
Researchers from Huawei Noah's Ark Lab, The Hong Kong University of Science and Technology, and Southern University of Science and Technology developed Mixture of Cluster-Conditional LoRA Experts (MoCLE) to mitigate task conflicts in large vision-language model instruction tuning. MoCLE consistently outperformed baseline InstructBLIP, achieving up to +3.9% on Visual Spatial Reasoning and +2.9% on IconQA, by intelligently routing instructions to specialized LoRA experts complemented by a universal expert.
Key insights
- Instruction embeddings provide an effective, automatic means to partition diverse visual-language tasks, grouping similar tasks and separating conflicting ones.
- Explicitly conditioning expert routing on instruction clusters can effectively mitigate negative transfer in multi-task LVLM instruction tuning.
- A "universal expert" that sees all data is crucial for maintaining broad generalization, complementing specialized experts which, by design, see less diverse data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.