Reproduce “CyberHost: Taming Audio-driven Avatar Diffusion Model with Region Codebook Attention” on OpenResearch

About this paper

Gaojie Lin, Jianwen Jiang, Chao Liang, Tianyun Zhong, Jiaqi Yang, Yanbo Zheng

Researchers from ByteDance and Zhejiang University introduced CYBERHOST, a one-stage diffusion framework capable of directly generating expressive half-body human videos from a single image and audio input. This framework achieves substantially lower FID (32.97 vs. 95.45) and FVD (555.8 vs. 1185.8) compared to existing two-stage methods, demonstrating robust zero-shot generalization and superior local detail preservation across various animation tasks.

Key insights

• Directly integrating explicit attention to critical local regions like faces and hands, via a Region Attention Module, substantially improves detail synthesis and identity consistency in generative models.
• Overcoming motion uncertainty in full-body generation requires embedding human structural priors and motion control signals directly into the diffusion process, rather than relying solely on ambiguous audio cues.
• A single, unified diffusion framework can effectively handle complex audio-to-video tasks for the human body, simplifying the pipeline and improving overall generation efficiency and quality compared to multi-stage systems.

Summary by alphaXiv.