Reproduce “Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zeren Chen, Ziqin Wang, Zhen Wang, Huayang Liu, Zhenfei Yin, Si Liu, Lu Sheng, Wanli Ouyang and 2 more
Octavius is a framework for multimodal large language models (MLLMs) that addresses task interference by integrating a novel LoRA-MoE decoder, achieving an average performance improvement of approximately 20% compared to non-MoE baselines across diverse 2D and 3D vision-language tasks. This approach leverages instance-based gate routing to dynamically activate specialized LoRA experts for specific tasks, and includes a specialized 3D point cloud encoder.
Key insights
- Integrating Mixture-of-Experts (MoE) with LoRA via an instance-based gate effectively mitigates task interference in MLLMs, allowing specialized pathways for different tasks.
- An instance-based gate routing strategy, which leverages the input question to select experts, is more effective and generalizable for MLLM tasks than conventional token-based or dense gating.
- Extracting 3D features by focusing on object-level regions and aggregating them into scene-level representations, aligned with language and image modalities, is critical for robust 3D understanding in MLLMs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.