Reproduce “SMIR: Efficient Synthetic Data Pipeline To Improve Multi-Image Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Andrew Li, Rahul Thapa, Rahul Chalamala, Qingyang Wu, Kezhen Chen, James Zou
The SMIR project introduces an efficient synthetic data pipeline and a new evaluation benchmark to enhance multi-image reasoning in open-source Vision-Language Models. It generated 160,000 synthetic training conversations, each with an average of 4.65 images and 9.65 turns, at up to 50 times lower cost and 10 times faster speed than closed-source methods. Models fine-tuned with SMIR data demonstrated an 8% improvement on the SMIR-BENCH, compared to existing instruction-tuned models.
Key insights
- Fusing image and caption embeddings effectively captures multimodal relationships, enabling better identification and clustering of genuinely correlated images for more complex reasoning tasks.
- Leveraging efficient, open-source LLMs can drastically reduce the cost (up to 50x) and increase the speed (up to 10x) of generating high-quality synthetic multi-image instruction data, making it accessible for the open-source community.
- Multi-turn, open-ended evaluation frameworks, judged by advanced VLMs, provide a more rigorous and comprehensive assessment of a model's expressiveness and reasoning justifications compared to traditional benchmarks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.