Reproduce “Sound to Visual Scene Generation by Audio-to-Visual Latent Alignment” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kim Sung-Bin, Arda Senocak, Hyunwoo Ha, Andrew Owens, Tae-Hyun Oh
Sound2Scene, developed by researchers including those from POSTECH, enables the generation of high-quality, diverse visual scenes directly from unrestricted audio by aligning audio features to a pre-trained visual latent space and leveraging a module for selecting highly correlated audio-visual pairs. The system can synthesize scenes from various real-world sounds and offers control over generated visuals through manipulations of input audio properties.
Key insights
- Bridging the modality gap between audio and visual signals can be effectively achieved by aligning audio features to a fixed, pre-trained visual latent space.
- Utilizing contrastive learning, specifically symmetric InfoNCE loss, proves robust for self-supervised alignment of heterogeneous modalities in the absence of explicit labels.
- Intelligent selection of highly correlated audio-visual training pairs, through sound source localization, significantly improves learning robustness and addresses issues of incongruent correlations in real-world video data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.