Reproduce “VTI-CoT: Visual-Textual Interleaved Chain of Thought for Video Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shufan Zhang, Ziyue Lin, Bairun Wang, Lei Jin, Xuanding Ding, Xinzhu Ma, Kunlin Yang
VTI-CoT proposes a framework that integrates visual information directly into each step of a Chain-of-Thought reasoning process for video understanding. It develops an automated pipeline to generate multimodal CoT data and uses an OCR-based rendering method to compress these sequences, yielding superior performance across six diverse video reasoning benchmarks while enhancing training efficiency.
Key insights
- Explicitly interleaving visual evidence with textual reasoning steps within a Chain-of-Thought framework significantly enhances video reasoning accuracy and interpretability by grounding inferences in visual data.
- Automated generation of high-quality visual-textual interleaved CoT datasets is feasible and effective, overcoming data scarcity for fine-grained multimodal supervision.
- Contextual Optical Compression (OCR) can efficiently compress long multimodal reasoning sequences into fixed-token visual signals, substantially improving training efficiency and stability without compromising performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.