Reproduce “AUGlasses: Continuous Action Unit based Facial Reconstruction with Low-power IMUs on Smart Glasses” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yanrong Li, Tengxiang Zhang, Xin Zeng, Yuntao Wang, Haotian Zhang, Yiqiang Chen
Researchers at the Institute of Computing Technology, Chinese Academy of Sciences, and Tsinghua University introduced AUGlasses, a low-power system integrated into smart glasses that uses Inertial Measurement Units (IMUs) for continuous, privacy-preserving facial reconstruction. The system achieves real-time 3D facial animation with an average NME of 2.75% while consuming only 49.95 mW.
Key insights
- Subtle skin deformations in the temporal muscle area, detected by low-power IMUs, provide sufficient information for continuous and detailed facial expression inference.
- The hybrid CNN-Transformer model, combined with specialized signal processing for artifact removal and a novel training strategy, can effectively translate minimal, non-visual sensor data into accurate facial action unit predictions.
- The system exhibits strong cross-user generalizability for AU estimation and 3D reconstruction, and its low power consumption makes continuous, privacy-preserving facial tracking viable for daily-wear smart glasses.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.