Reproduce “Skeleton Sequence and RGB Frame Based Multi-Modality Feature Fusion Network for Action Recognition” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaoguang Zhu, Ye Zhu, Haoyu Wang, Honglin Wen, Yan Yan, Peilin Liu
A Multi-Modality Feature Fusion (MMFF) network is presented for action recognition, combining skeleton sequences with a single RGB frame using attention mechanisms. The method achieves competitive accuracy on datasets like NTU RGB+D, reducing computational parameters by up to 59.4% and FLOPs by up to 88.6% compared to approaches relying on full RGB videos.
Key insights
- A single, strategically chosen RGB frame can provide sufficient spatial context to complement skeleton sequences for robust action recognition, thereby significantly reducing computational overhead without a proportional drop in accuracy.
- Attention mechanisms, particularly an early fusion module guided by skeleton dynamics, are crucial for effectively transferring temporal knowledge from skeleton data to the spatial features of the RGB frame.
- Targeted pre-processing, such as projecting 3D skeleton data to crop 2D RGB frames, enhances the RGB stream's focus on relevant human-object interactions, improving both accuracy and efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.