Reproduce “Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Weiqi Feng, Yangrui Chen, Shaoyu Wang, Yanghua Peng, Haibin Lin, Minlan Yu
Optimus is a distributed training system developed by researchers from Harvard University and Bytedance that minimizes GPU idle time in large-scale Multimodal Large Language Model (MLLM) training by strategically scheduling encoder computations within the LLM backbone's idle periods. This approach accelerates MLLM training by 20.5%-21.3% on 3072 GPUs, achieving an average of 20.3% performance improvement over existing systems.
Key insights
- Significant GPU idle times (bubbles) in MLLM training, primarily within the LLM backbone's forward and backward passes, can be repurposed for encoder computations.
- Effective MLLM optimization requires separate, yet coordinated, parallelism strategies for heterogeneous components (encoders and LLM) across all GPUs.
- Exploiting fine-grained, sub-millisecond GPU bubbles demands kernel-level scheduling to maximize utilization, rather than coarser layer-level approaches.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.