Reproduce “Planning with Sketch-Guided Verification for Physics-Aware Video Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yidong Huang, Zun Wang, Han Lin, Dong-Ki Kim, Shayegan Omidshafiei, Jaehong Yoon, Yue Zhang, Mohit Bansal
SketchVerify introduces a training-free framework for physics-aware video generation, iteratively refining motion plans using lightweight video sketches and a multimodal verifier. This approach achieves substantially more physically plausible and temporally consistent video outputs while being significantly more efficient than prior iterative refinement methods.
Key insights
- Decoupling motion plan verification from full video synthesis via lightweight video sketches allows for efficient test-time iterative refinement, providing a nearly 93% speed-up over full video-based iterative methods.
- Multimodal verification (visual input + language) is crucial for accurately assessing both semantic instruction following and fine-grained physical plausibility, outperforming language-only verifiers.
- Iterative sampling and scoring of candidate trajectories by MLLM planners and verifiers enables the selection of dynamically coherent and physically consistent motion plans prior to expensive pixel-level diffusion.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.