Reproduce “Video-as-Answer: Predict and Generate Next Video Event with Joint-GRPO” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junhao Cheng, Liang Hou, Xin Tao, Jing Liao
Researchers from City University of Hong Kong and Kuaishou Technology developed VANS, a model for Video-Next-Event Prediction (VNEP) that generates dynamic video answers to contextual questions, moving beyond text-only responses. Utilizing a novel Joint-GRPO reinforcement learning strategy to align a VLM and VDM, VANS demonstrates superior performance in generating semantically accurate and visually coherent videos, achieving a lower Fréchet Video Distance (78.32) and higher CLIP-V (0.8021) on procedural benchmarks compared to existing methods.
Key insights
- Shifting from textual to video-based answers (VNEP) fundamentally enhances AI's utility for tasks requiring visual demonstration, such as procedural learning and "how-to" guides.
- Overcoming the semantic-to-visual misalignment between reasoning (VLM) and generation (VDM) is crucial for producing coherent and faithful video responses.
- A two-stage reinforcement learning strategy, Joint-GRPO, effectively co-adapts the VLM and VDM, enabling them to mutually reinforce each other's capabilities and bridge the cross-modal gap.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.