Reproduce “You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shengkun Tang, Yaqing Wang, Zhenglun Kong, Tianchi Zhang, Yao Li, Caiwen Ding, Yanzhi Wang, Yi Liang and 1 more
A dynamic early exiting strategy, MuE, improves the inference efficiency of unified Transformer-based vision-language models by adaptively skipping layers based on input complexity. This method achieved up to 50% computation reduction for visual entailment and 40.2% for image captioning, maintaining strong task performance.
Key insights
- Computational requirements for large unified VLMs vary significantly across inputs, making dynamic early exiting a viable strategy for efficiency.
- Layer-wise similarity of hidden states can reliably indicate representational saturation, offering a robust, confidence-independent criterion for early exiting in both encoders and decoders.
- Modality-specific processing within the encoder allows for more precise and efficient computation by recognizing the differential importance of image and text for various VL tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.