Reproduce “Hyper-Bagel: A Unified Acceleration Framework for Multimodal Understanding and Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yanzuo Lu, Xin Xia, Manlin Zhang, Huafeng Kuang, Jianbin Zheng, Yuxi Ren, Xuefeng Xiao
ByteDance Seed developed Hyper-Bagel, a unified framework designed to accelerate both multimodal understanding and generation in models like BAGEL. The framework achieves a 2.16x speedup for understanding and up to 22x inference speedup for generation, maintaining or improving output quality compared to baseline models.
Key insights
- Aggregating comprehensive features from all target layers via meta queries significantly improves a lightweight draft model's ability to predict multimodal tokens accurately for speculative decoding.
- Zero-initializing specific layers and adding residual connections during draft model training provides a more stable learning objective, crucial for effectively modeling target model features from diverse embedding spaces.
- A multi-stage diffusion distillation pipeline, including distribution matching via ODE (DMDO) and VLM-based reward feedback (ReFL), enables substantial NFE reduction while maintaining high-fidelity and semantically aligned multimodal generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.