Reproduce “LAV: Audio-Driven Dynamic Visual Generation with Neural Compression and StyleGAN2” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jongmin Jung, Dasaem Jeong
The LAV system generates dynamic visual content from audio by mapping latent embeddings from a neural audio compression model (EnCodec) to StyleGAN2's style space via a randomly initialized linear projection. Developed by researchers at Sogang University, this approach produces high-quality, semantically coherent visuals with temporal smoothness without requiring task-specific training.
Key insights
- Pre-trained neural audio compression models, such as EnCodec, contain semantically rich latent representations suitable for creative audio-visual applications.
- Effective audio-visual mapping can be achieved with a simple, randomly initialized linear projection layer, challenging the assumption that task-specific training is always necessary.
- Hierarchical style smoothing allows for maintaining global visual coherence while ensuring fine details remain dynamic and responsive to audio changes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.