Reproduce “Video Joint-Embedding Predictive Architectures for Facial Expression Recognition” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lennart Eing, Cristina Luna-Jiménez, Silvan Mertes, Elisabeth André
This paper introduces Video Joint-Embedding Predictive Architectures (V-JEPAs) for Facial Expression Recognition (FER), demonstrating that an embedding-based pre-training approach outperforms traditional pixel-level reconstruction methods on several benchmarks. Using a frozen V-JEPA encoder, the proposed method achieves state-of-the-art Unweighted Average Recall on RAVDESS and significantly improved Weighted Average Recall on CREMA-D for vision-only FER.
Key insights
- Pixel-level reconstruction pretext tasks are not necessary for achieving state-of-the-art performance in Facial Expression Recognition; embedding-based prediction is a viable and effective alternative.
- Video Joint-Embedding Predictive Architectures (V-JEPAs) learn more abstract, semantic, and task-relevant representations by focusing on predictive embeddings, which are more beneficial for high-level tasks like emotion recognition than raw pixel details.
- Leveraging a frozen pre-trained V-JEPA encoder with only shallow classifiers significantly reduces computational cost for FER, while demonstrating the high transferability and quality of the learned representations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.