Reproduce “ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jieyu Zhang, Le Xue, Linxin Song, Jun Wang, Weikai Huang, Manli Shu, An Yan, Zixian Ma and 6 more
Researchers from the University of Washington and Salesforce Research developed PROVISION, a programmatic system for generating over 10 million high-quality, vision-centric instruction data points for multimodal language models. This data significantly improved MLM performance on various vision-centric benchmarks, showing up to an 8% increase in specific tasks.
Key insights
- Programmatic data generation offers enhanced interpretability, factual accuracy, scalability, and cost-effectiveness compared to purely generative model-based approaches.
- While data derived from human-annotated scene graphs generally performs better, large quantities of model-generated scene graph data can achieve comparable performance for single-image tasks.
- Optimal MLM performance is achieved through a synergistic approach, incorporating PROVISION data in both pre-training and instruction tuning stages, and utilizing a mix of multiple-choice and short-answer formats.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.