Reproduce “Smooth-Foley: Creating Continuous Sound for Video-to-Audio Generation Under Semantic Guidance” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yaoyun Zhang, Xuenan Xu, Mengyue Wu
Researchers at Shanghai Jiao Tong University developed Smooth-Foley, a video-to-audio generation model that synthesizes continuous, semantically and temporally aligned audio for videos with moving objects. The model improves upon existing methods by integrating high-resolution frame-wise video conditioning and semantic-enhanced temporal guidance, demonstrating superior performance in generating realistic and synchronized continuous sounds.
Key insights
- The use of frame-wise video embeddings provides high-resolution temporal conditioning, improving precise audio-visual alignment.
- Semantic-enhanced temporal guidance, integrating textual labels with visual features, helps the model accurately associate sounds with visual events.
- The model specifically targets continuous sound generation through its architecture and training on the VGGSound-Continuous dataset, addressing limitations of prior V2A methods.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.