Reproduce “Demo-ICL: In-Context Learning for Procedural Video Knowledge Acquisition” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuhao Dong, Shulin Tian, Shuai Liu, Shuangrui Ding, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Jiaqi Wang and 1 more
Demo-ICL tests whether video models can apply supplied procedures to instructional videos through text instructions, related video demonstrations, or candidate selection. The 7B model uses video instruction tuning and information-assisted preference optimization, and is evaluated on 1,200 questions. With demonstrations, it scores 43.4% on text and 32.0% on video; the benchmark covers instructional videos, and video transfer remains difficult.
Key insights
- State-of-the-art proprietary MLLMs like Gemini-2.5-Pro (e.g., 36.2% for Video-demo ICL) and GPT-4o show low accuracy on Demo-ICL-Bench, revealing a substantial gap between current MLLMs and human-level (e.g., 80.4% for Video-demo ICL) understanding and adaptation from demonstrations.
- The effectiveness of in-context learning is highly dependent on model scale, with larger models demonstrating significantly greater performance gains (over 20 points for Qwen2.5-VL-72B) when provided with text demonstrations.
- Information-assisted Direct Preference Optimization (DPO), which leverages automatically generated assistive information, is particularly effective in providing richer feedback and improving response quality for complex demo-driven ICL tasks compared to vanilla DPO.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.