Reproduce “Jump Cut Smoothing for Talking Heads” on OpenResearch

About this paper

Xiaojuan Wang, Taesung Park, Yang Zhou, Eli Shechtman, Richard Zhang

A framework is presented to eliminate visually jarring jump cuts in talking head videos by synthesizing realistic intermediate frames. It leverages DensePose keypoints and a cross-modal attention mechanism to handle significant motion changes and preserve identity, outperforming state-of-the-art frame interpolation methods in photorealism and identity preservation, especially for large head rotations.

Key insights

• Mid-level semantic motion representations, like DensePose combined with facial landmarks, are more robust than optical flow for handling large, complex motion changes in talking head videos.
• A cross-modal attention mechanism can effectively aggregate and select relevant appearance features from diverse source frames, improving synthesis quality and identity preservation.
• Training with simulated incomplete keypoint scenarios and employing a recursive filling strategy enables the generator to inpaint missing regions and produce coherent intermediate frame sequences for jump cut transitions.

Summary by alphaXiv.