Reproduce “Fostering Video Reasoning via Next-Event Prediction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haonan Wang, Hongfu Liu, Xiangyan Liu, Chao Du, Kenji Kawaguchi, Ye Wang, Tianyu Pang
This paper introduces Next-Event Prediction (NEP), a new learning task designed to enhance multimodal large language models' ability to perform temporal reasoning by predicting future events from past video observations. It also presents V1-33K, a large-scale, automatically generated dataset, and FutureBench, a benchmark to evaluate this capability, demonstrating that NEP improves temporal understanding without degrading general video comprehension.
Key insights
- Deductive reasoning, as implemented through Next-Event Prediction, performs better for temporal understanding compared to inductive (video Q&A) or abductive (previous-event prediction) reasoning.
- Models trained with Next-Event Prediction enhance their temporal reasoning abilities while preserving performance on general video understanding tasks.
- Supervised Fine-Tuning (SFT) is an effective and reliable training strategy for Next-Event Prediction, whereas scaling training data beyond 5,000 samples does not consistently improve performance and can sometimes lead to degradation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.