Reproduce “Audio- and Gaze-driven Facial Animation of Codec Avatars” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Alexander Richard, Colin Lea, Shugao Ma, Juergen Gall, Fernando de la Torre, Yaser Sheikh

The authors present a system that drives photorealistic 3D facial animation from audio and gaze. It uses separate encoders and learned, time-varying weights to combine the signals in a shared representation and predict facial coefficients. In a reported comparison, it achieves lower all-landmark error (13.36 versus 15.52) and higher lip-closure F1 (0.546 versus 0.462) than conventional regression, within an evaluation on five hours from three people.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.