Reproduce “Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang and 2 more
Researchers from Zhejiang University, National University of Singapore, and Nanyang Technological University developed the VPG-C module and a synthetic training strategy to empower Multimodal Large Language Models (MLLMs) with enhanced understanding of detailed, interleaved visual and textual instructions. This approach significantly improves MLLM performance on a new benchmark, DEMON, by fostering meticulous discernment of subtle visual details and reasoning-aware attention in a zero-shot setting.
Key insights
- Current MLLMs exhibit an "inductive bias" in their Visual Prompt Generators, causing them to neglect fine-grained visual details crucial for complex, interleaved instructions.
- Intermediate representations within the LLM can be effectively leveraged to generate instruction-specific guidance for retrieving complementary visual details, mimicking a "think twice" mechanism.
- Synthetic discriminative training, which uses generated images with subtle differences in low-attention areas, offers a data-efficient method to instill reasoning-aware detail perception in MLLMs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.