Reproduce “LS-HAR: Language Supervised Human Action Recognition with Salient Fusion, Construction Sites as a Use-Case” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mohammad Mahdavian, Mohammad Loni, Ted Samuelsson, Mo Chen
LS-HAR combines RGB video and 3D skeleton sequences for human action recognition, using conditional language supervision to refine global and body-part-specific skeleton representations before salient multimodal fusion. It reaches 93.7% on NTU-RGB+D Cross-Subject, 97.3% Cross-View, 90.3% on NTU-RGB+D 120 Cross-Subject, and 96.4% on the proposed VolvoConstAct construction-action dataset.
Key insights
- Language supervision mainly benefits the skeleton branch: skeleton-only accuracy on NTU60 Cross-Subject rises from 86.2% to 86.6%, whereas adding language supervision to both modalities slightly reduces combined performance.
- Body-part-specific semantic guidance helps the model focus on action-relevant regions, such as the active arm during waving, rather than relying only on global pose information.
- Salient feature reduction improves fusion efficiency and accuracy; removing it lowers performance by roughly 0.8 percentage points, while learnable and skeleton-conditioned prompts provide smaller additional gains.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.