Reproduce “AutoFoley: Artificial Synthesis of Synchronized Sound Tracks for Silent Videos with Deep Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sanchita Ghose, John J. Prevost
AutoFoley from the University of Texas at San Antonio is a deep learning system that automatically generates realistic, synchronized Foley sound effects for silent videos. It produces sound tracks that human evaluators often perceive as original recordings, outperforming prior audio-visual synthesis methods in accuracy and quality for clean sound effects.
Key insights
- The Automatic Foley Dataset (AFD) with clean, distinct sound effects is crucial for training a system to generate professional-quality Foley effects, unlike general audio-visual datasets.
- Interpolation of video frames effectively preserves smooth motion details, which are essential for accurate sound prediction and managing audio-video sampling rate mismatches.
- Frame-Sequence Networks (Fast-Slow LSTM) offer superior overall performance in sound retrieval, while Frame-Relation Networks (Temporal Relation Networks) excel at action-sensitive sound categories by capturing temporal relationships between frames.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.