Reproduce “Semantic-Critical Regions for Video-to-Audio Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yun Fang, Wei Guo, Zhuonan Liang, Heng Wang, Weidong Cai
SCR-V2A generates sounds for silent videos by combining the original scene with Semantic-Critical Regions (SCRs), heatmaps of visual evidence for predicted sound events. A shared video encoder preserves scene context while a control adapter guides an audio diffusion model. On VGGSound, it reports FAD 1.07, versus 1.84 for MMAudio; this indicates better feature-distribution matching on that benchmark, not playback quality.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.