Reproduce “Cross-view Action Recognition Understanding From Exocentric to Egocentric Perspective” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Thanh-Dat Truong, Khoa Luu

A Cross-View learning approach (CVAR) from the University of Arkansas models geometric correlations within Transformer self-attention to effectively transfer knowledge from large exocentric video datasets to egocentric ones for action recognition. This method, operating without paired cross-view data, established new state-of-the-art results on benchmarks including Charades-Ego (31.95% mAP) and EPIC-Kitchens-55 (73.52% Top 1 Verb accuracy).

Key insights

  • A linear proportional relationship exists between the cross-view correlation of video content and the cross-view correlation of attention maps, guided by camera transformations.
  • Minimizing the Unpaired Cross-View Self-Attention Loss on unpaired data can effectively impose the geometric correlation constraint on implicitly paired samples.
  • Applying the cross-view loss to deeper Transformer layers yields greater performance improvements, indicating that these layers benefit most from geometric regularization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.