Reproduce “AVENUE: Audio-Video EditiNg Understanding and Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hayeon Kim, Yoojin Jang, Jaejun Yoo
AVENUE introduces a comprehensive benchmark and evaluation framework for audio-video editing, designed to assess models' ability to perform modality-selective edits while preserving untargeted content. The framework, leveraging human-verified annotations and an MLLM-as-a-judge, reveals that current AV editing models frequently introduce unintended changes in non-targeted modalities, highlighting a persistent challenge in controllable multimodal generation.
Key insights
- Existing AV editing models consistently struggle with modality-selective controllability, frequently introducing unintended changes in non-targeted modalities regardless of their architectural paradigm.
- No single architectural paradigm (joint, sequential, separate) offers consistent superiority across all AV editing tasks, with their effectiveness varying based on the specific edit type and modality target.
- Inversion-based audio editing approaches (e.g., ZETA) demonstrate a promising balance between achieving high edit accuracy and preserving untargeted audio content, offering insights for future model design.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.