Reproduce “AC-Foley: Reference-Audio-Guided Video-to-Audio Synthesis with Acoustic Transfer” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Pengjun Fang, Yingqing He, Yazhou Xing, Qifeng Chen, Ser-Nam Lim, Harry Yang
AC-FOLEY, developed by researchers at The Hong Kong University of Science and Technology, introduces a reference-audio-guided framework for video-to-audio synthesis, enabling fine-grained acoustic control. The system demonstrates superior performance over existing methods in distribution matching, semantic alignment, and spectral fidelity, with human evaluations showing an 83.5% preference for its acoustic fidelity.
Key insights
- Directly conditioning on detailed acoustic features extracted via a VAE encoder from a reference audio clip enables more precise control over sound characteristics than using semantic-only encoders.
- A two-stage training regimen, combining overlapping and non-overlapping audio conditioning, is critical for teaching models to leverage acoustic self-similarity for robust feature transfer and temporal adaptation without simply copying the reference.
- Integrating multimodal cues from video, text, and synchronization mechanisms with fine-grained acoustic features through adaptive layer normalization is necessary for achieving high-fidelity, synchronized, and controllable audio generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.