Reproduce “STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haiyi Qiu, Minghe Gao, Long Qian, Kaihang Pan, Qifan Yu, Juncheng Li, Wenjie Wang, Siliang Tang and 2 more
Researchers from Zhejiang University and NUS developed STEP, a graph-guided self-training framework that enhances Video-LLMs' ability to perform multi-step compositional reasoning by autonomously generating rationale-rich data. The method achieved a 21.3% improvement on tasks requiring three or more reasoning steps on the STAR dataset.
Key insights
- Structured Spatio-Temporal Scene Graphs effectively abstract multi-granular video semantics, providing a robust foundation for autonomously generating complex compositional reasoning tasks.
- Self-training with explicitly supervised, graph-guided rationales is more effective for enhancing compositional reasoning capabilities than relying on larger quantities of traditional answer-only instruction-tuning data.
- A balanced distribution of questions across different reasoning steps (1-step, 2-step, ">=3-step") and equal weighting of answer and rationale loss are crucial for optimizing model performance in multi-step reasoning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.