Reproduce “VidFormer: A novel end-to-end framework fused by 3DCNN and Transformer for Video-based Remote Physiological Measurement” on OpenResearch

About this paper

Jiachen Li, Shisheng Guo, Longzhen Tang, Cuolong Cui, Lingjiang Kong, Xiaobo Yang

Researchers at the University of Electronic Science and Technology of China developed VidFormer, an end-to-end framework that integrates 3DCNNs and Transformers for video-based remote physiological measurement. This hybrid architecture consistently achieved state-of-the-art accuracy in heart rate, heart rate variability, and respiratory rate estimation across diverse public datasets, demonstrating improved robustness against environmental noise and subject variability.

Key insights

• Synergistically combining 3DCNNs and Transformers within a dual-branch architecture effectively mitigates the limitations of each, balancing local detail extraction with global context understanding for improved rPPG signal reconstruction.
• Explicitly designed interaction modules (CTIM) are crucial for robust performance, enabling the transfer of inductive bias from CNNs to Transformers and global information from Transformers to CNNs, particularly aiding convergence and generalization.
• Disentangling spatial and temporal attention mechanisms within both convolutional (GA-3DCNN) and Transformer (ST-MHSA) blocks allows the model to more effectively capture both inter-region spatial relationships and the periodic dynamics of physiological signals.

Summary by alphaXiv.