Reproduce “Blendshapes GHUM: Real-time Monocular Facial Blendshape Prediction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ivan Grishchenko, Geng Yan, Eduard Gabriel Bazavan, Andrei Zanfir, Nikolai Chinaev, Karthik Raveendran, Matthias Grundmann, Cristian Sminchisescu
Google's Blendshapes GHUM system provides real-time facial animation on mobile devices by predicting facial blendshape coefficients from a single RGB camera. The system enables expressive facial mirroring onto avatars, with the complete pipeline running at over 30 frames per second on modern smartphones, and individual components like blendshape prediction operating at 1.2ms per frame on a Pixel 6.
Key insights
- Blendshapes provide an interpretable, artistically controllable, and efficient representation for facial animation, compatible with existing industry workflows.
- A dual-loss training strategy, combining coefficient and reconstructed landmark losses, ensures both numerical accuracy of predicted blendshapes and visual fidelity of the resulting expressions.
- A large, diverse dataset of 3D facial scans with corresponding blendshape coefficients, combined with lightweight neural network architectures, enables real-time, high-quality facial animation on mobile devices.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.