Reproduce “KeyFrame-Compass: Towards Comprehensive Evaluation of Keyframe-Conditioned Video Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuqi Tang, Tengfei Liu, Yizheng Lai, Yuran Wang, Yang Shi, Wanshun Su, Zhuoran Zhang, Qixun Wang and 13 more
A new benchmark, KeyFrame-Compass, enables comprehensive, diagnostic evaluation of keyframe-conditioned video generation models by assessing both accurate keyframe reproduction and overall video quality. Its application revealed a consistent trade-off between keyframe fidelity and natural video synthesis, as well as performance degradation with increased constraint density and specific challenges for open-source models in interpreting storyboard-grid inputs.
Key insights
- Models face an inherent trade-off between precisely reproducing input keyframes and generating natural, fluent video transitions, often resulting in a "slide-show-like" effect for high fidelity.
- Instruction adherence, particularly for visual and temporal control, degrades consistently as the density of keyframe constraints increases.
- Open-source models struggle significantly with interpreting storyboard-grid inputs, frequently failing to decompose them into coherent, temporally ordered keyframe sequences.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.