Reproduce “DVANet: Disentangling View and Action Features for Multi-View Action Recognition” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Nyle Siddiqui, Praveen Tirupattur, Mubarak Shah

DVANet presents a framework for multi-view action recognition that disentangles view-specific and action-relevant features using only RGB video data. This model achieved new state-of-the-art unimodal performance on benchmark datasets including NTU RGB+D, NTU RGB+D 120, PKU-MMD, and N-UCLA.

Key insights

  • Explicitly disentangling view and action features through dedicated architectural components and supervised loss functions is crucial for achieving superior multi-view action recognition performance with RGB data alone.
  • Learnable action and view queries within a transformer decoder effectively capture and separate the respective semantic information.
  • A combination of cross-entropy, supervised contrastive losses, and an orthogonality constraint enables the model to learn robust action representations that are invariant to viewpoint changes.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.