Reproduce “Multimodal Skeleton-Based Action Representation Learning via Decomposition and Composition” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hongsong Wang, Heng Fei, Bingxuan Dai, Jie Gui
A decomposition-and-composition training framework learns multimodal skeleton representations with a single shared encoder at inference time. By preserving modality-specific information from joint, bone, and motion streams while matching a late-fused training target, the method improves over the unified UmURL baseline across NTU RGB+D and PKU-MMD II, reaching 85.8% cross-subject and 91.8% cross-view accuracy on NTU-60 at 2.54 GFLOPs and 114.1 FPS.
Key insights
- Decomposition and composition address complementary needs: decomposition retains distinctive unimodal information, whereas composition supplies direct guidance for the fused representation.
- Separating spatial and temporal branches yields stronger supervision than applying a single global alignment objective; composition alone performs substantially worse in ablations.
- A strong joint-only representation can approach or surpass some multimodal alternatives, while adding complementary modalities generally improves performance, though motion is less reliable on certain PKU-MMD II combinations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.