Reproduce “Emo-LiveSync: Real-Time Universal Lip Synchronization with Explicit Emotion Control” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sangjoon Yu, Seungeun Lee, JoonHeyok Shin, Ji-Su Kang

Emo-LiveSync edits mouths in video to match new audio while retaining appearance. It predicts implicit keypoints from audio, aligns them with motion markers extracted by a pretrained video model, and warps frames instead of regenerating them. An optional gate adds expression offsets. On UniVox-Bench, it scores better on synchronization; the benchmark covers 34 subjects and seven languages, with emotion disabled in quantitative tests.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.