Reproduce “Synthetic Stimuli, Real Gains: Rethinking VLM Fine-Tuning Through Fully Controlled Data Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Massimo Rizzoli, Simone Alghisi, Seyed Mahed Mousavi, Giuseppe Riccardi
Researchers at the University of Trento developed a methodology for fine-tuning Vision-Language Models using fully controlled synthetic data, achieving greater than 96% accuracy on synthetic spatial reasoning tasks with minimal data and improving transfer performance on real-world COCO images by over 20 percentage points for certain models. This approach demonstrates that high-quality, balanced synthetic datasets can foster robust visual reasoning and outperform fine-tuning with large, potentially biased real-world data.
Key insights
- VLMs exhibit strong inherent spatial biases before fine-tuning, often misinterpreting locations in regions like the center-left/right, which controlled synthetic data can effectively mitigate.
- Data quality, balance, and rigorous control are more impactful than sheer data quantity for VLM fine-tuning, with large, biased real-world datasets shown to degrade performance.
- Moderate exposure to visual distractors during synthetic fine-tuning can enhance robustness and improve transferability to more complex real-world scenes for encoder-decoder models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.