Reproduce “OpenFace 3.0: A Lightweight Multitask System for Comprehensive Facial Behavior Analysis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiewen Hu, Leena Mathur, Paul Pu Liang, Louis-Philippe Morency
OPENFACE 3.0 combines facial landmark detection, action-unit detection, gaze estimation, and emotion recognition in an open-source toolkit built around shared facial representations. It reports competitive benchmark performance across these tasks, while processing a frame in 38 ms on CPU with 29.4 million parameters; performance gains vary by task, and emotion recognition is more robust to large face angles in the reported evaluation.
Key insights
- Multitask learning improved AU and gaze results over single-task training in the reported ablation, but ordinary multitask training reduced emotion accuracy; uncertainty weighting partly mitigated that trade-off.
- Shared task representations may help with angled faces: on faces rotated more than 45 degrees, OPENFACE 3.0 achieved 58.64% emotion accuracy, above the two reported comparison systems.
- An integrated toolkit can reduce duplicated inference work, but competitive overall performance does not mean it surpasses specialist models on every task.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.