Reproduce “Feature Hallucination for Self-supervised Action Recognition” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Lei Wang, Piotr Koniusz

This research introduces a self-supervised multimodal framework for action recognition that addresses incomplete data and inefficient feature fusion. It integrates novel Object Detection and Saliency Detection Features, a hallucination mechanism for missing modalities, and uncertainty modeling, yielding improved accuracy and robustness across various datasets, including up to a 6% mAP gain on Charades.

Key insights

  • Domain-specific descriptors like ODF and SDF provide valuable explicit spatial and motion cues, enriching feature representations and improving performance, especially in fine-grained action recognition.
  • Self-supervised hallucination drastically reduces inference-time computational overhead by generating expensive auxiliary features 'for free,' enabling robust multimodal AR even with incomplete data.
  • Modeling aleatoric uncertainty in feature learning enhances the model's robustness to noisy or unreliable auxiliary data, leading to more stable and confident predictions across diverse video sources.
  • Well-designed 'handcrafted' descriptors (BoW/FV, OFF) can still provide performance boosts even when integrated with powerful self-supervised video foundation models like VideoMAE V2 and InternVideo2.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.