Reproduce “HeyGen Voice: Scene-Aware Voice Cloning for Talking Videos” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Pavel Andreev, Andrei Gusev, Adithya Narayan, Yi Ren, Ivan Somov, Zijian Li, Sergey Demyanov, Viktor Goriachko and 10 more
HeyGen Voice separates speaker identity from scene sound. It uses a reference for speaker cues; a target image and descriptions guide delivery and acoustics. Users choose whether to retain reference sound or adapt to the scene. In a 50-scene comparison, it received 65.3% of decided ambient-fit votes against Seedance 2.5, but showed no clear advantage in preference or acoustic fit.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.