Reproduce “Instruction-Guided Visual Masking” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jinliang Zheng, Jianxiong Li, Sijie Cheng, Yinan Zheng, Jiaming Li, Jihao Liu, Yu Liu, Jingjing Liu and 1 more
Instruction-guided Visual Masking (IVM) introduces a versatile visual grounding model that automatically masks instruction-irrelevant image regions, allowing large multimodal models (LMMs) to focus on task-relevant visual content. The approach, leveraging a 1-million-sample dataset and a novel training framework, improved GPT4-V's accuracy on V*Bench by 26.2% and enhanced robotic control generalization.
Key insights
- Explicitly masking instruction-irrelevant visual regions significantly enhances LMMs' ability to align with complex instructions and reduces errors caused by visual noise.
- A sophisticated data generation pipeline, combining LLMs with an ensemble of visual grounding models, can efficiently create large-scale, high-quality, fine-grained visual grounding datasets.
- The DWSL framework effectively addresses the challenge of mixed-quality training data by weighting samples based on discriminator-assessed quality, leading to more robust model training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.