Reproduce “VEFX-Bench: A Holistic Benchmark for Generic Video Editing and Visual Effects” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiangbo Gao, Sicong Jiang, Bangya Liu, Xinghao Chen, Minglai Yang, Siyuan Yang, Mingyang Wu, Jiongze Yu and 7 more
Researchers from Texas A&M University, Visko Platform, and Abaka AI developed VEFX-Bench, a comprehensive benchmark for instruction-guided video editing, which includes a human-annotated dataset (VEFX-Dataset) and a specialized reward model (VEFX-Reward) that aligns closely with human judgments. Their evaluation of existing systems using VEFX-Bench revealed that current models often achieve high visual rendering quality but struggle with precise instruction following and ensuring only intended content is edited.
Key insights
- Video editing quality is multi-faceted, requiring decoupled assessment of instruction following, rendering quality, and edit exclusivity to provide granular insights into model performance beyond a single scalar score.
- Specialized reward models, jointly processing original video, instruction, and edited result, significantly outperform generic Vision-Language Models and prior reward models in accurately assessing video editing quality.
- Existing video editing systems consistently demonstrate a gap where visual plausibility (high Rendering Quality) is achieved, but reliable instruction following and precise content preservation (Edit Exclusivity) remain persistent weaknesses.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.