Reproduce “Multimodal Semantic Communication for Generative Audio-Driven Video Conferencing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haonan Tong, Haopeng Li, Hongyang Du, Zhaohui Yang, Changchuan Yin, Dusit Niyato
The Wav2Vid system, developed by researchers from Beijing University of Posts and Telecommunications, University of Hong Kong, Zhejiang University, and Nanyang Technological University, proposes a multimodal semantic communication system that exploits audio-video correlation in talking-head videos. This system reduces video conferencing data transmission by up to 83% by selectively sending video only for significant head movements and generating the rest of the visual content from audio, demonstrating improved performance in low SNR conditions.
Key insights
- Semantic communication, by focusing on meaning rather than raw data, can significantly reduce data transmission requirements for video conferencing.
- There is a strong correlation between audio (speech) and video (lip movements) in talking-head videos, which can be leveraged to generate visual content from audio.
- Selectively transmitting video data only for significant head pose changes and generating the rest using audio can drastically improve bandwidth efficiency without sacrificing perceptual quality.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.