Reproduce “LiveFoley: Dynamic Real-Time Video-to-Audio Generation for Immersive Interactive Worlds” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yusheng Dai, Vladimir Iashin, Kangdi Wang, Yuxuan Jiang, Ilpo Viertola, Weiqiang Wang, Qiuhong Ke, Min Zhao and 1 more
LiveFoley generates live-video audio while retaining temporal context through a trailing sliding window and a denoiser using neighboring audio patches. On VGGSound, it reaches predicted desynchronization 0.48 versus offline MMAudio-L’s 0.47 with 0.08-second frontend look-ahead; measured end-to-end latency is 156 ms on one A800 GPU, though overall audio quality still trails offline systems.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.