Reproduce “Low-Resource Guidance for Controllable Latent Audio Diffusion” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zachary Novack, Zack Zukowski, CJ Carr, Julian Parker, Zach Evans, Josiah Taylor, Taylor Berg-Kirkpatrick, Julian McAuley and 1 more
Researchers from UC San Diego and Stability AI developed a low-resource guidance framework for controllable latent audio diffusion, enabling fine-grained control over aspects like intensity, pitch, and rhythm. The method employs selective Training-Free Guidance and lightweight Latent-Control Heads (LatCHs) that operate directly in the VAE's latent space, achieving orders of magnitude speedup and reduced VRAM usage while maintaining high audio quality.
Key insights
- Performing gradient-based guidance directly in the compressed latent space, rather than on decoded high-dimensional audio, drastically reduces computational demands for long-form audio generation.
- Strategically applying guidance only at relevant stages of the diffusion process (Selective TFG) improves efficiency and helps maintain the naturalness and quality of the generated audio.
- LatCHs, when trained with noise-conditioned latents (LatCH-B), provide an accurate and resource-efficient proxy for complex audio feature extractors, enabling practical inference-time control.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.