Reproduce “S2H-DPO: Hardness-Aware Preference Optimization for Vision-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nitish Shukla, Surgan Jandial, Arun Ross
Researchers at Michigan State University and Carnegie Mellon University developed S2H-DPO, a framework that enhances multi-image reasoning in Vision-Language Models (VLMs) by systematically generating preference data across three levels of complexity for Direct Preference Optimization (DPO) training. This method achieved an average 5.27% improvement on multi-image benchmarks for LLaVA-v1.5-7B while maintaining single-image performance, improving autonomous visual search and cross-image comparison.
Key insights
- A systematic, prompt-driven approach to generating multi-image preference data across a hierarchy of reasoning levels is more effective and model-agnostic than methods relying on model-specific attributes.
- Direct Preference Optimization (DPO) provides a stronger learning signal for complex multimodal reasoning tasks compared to standard Supervised Fine-Tuning (SFT).
- Flat training, where models are exposed to a mix of simple and hard tasks simultaneously, often outperforms incremental curriculum-based training for multi-image reasoning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.