Reproduce “ViPLO: Vision Transformer based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detection” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jeeseung Park, Jin-Woo Park, Jong-Seok Lee

ViPLO introduces a two-stage Human-Object Interaction (HOI) detection method that effectively integrates Vision Transformer (ViT) backbones with a Masking with Overlapped Area (MOA) module for precise regional feature extraction. The method also employs a pose-conditioned self-loop graph neural network for interaction classification, achieving 37.22 mAP on HICO-DET, surpassing previous state-of-the-art one-stage methods while maintaining computational efficiency.

Key insights

  • Vision Transformers can be effectively integrated into two-stage object detection pipelines with a specialized region feature extraction module like MOA, which accurately handles their patch-based processing for arbitrary regions.
  • Modeling human perception processes, specifically by incorporating overall pose for interactiveness and local joint features for interaction type recognition, significantly enhances the performance of graph-based HOI classification.
  • Two-stage HOI detection methods can surpass one-stage methods in performance, achieving state-of-the-art results while retaining their inherent advantages in computational efficiency and flexibility, by leveraging advanced backbones and perception-inspired interaction classifiers.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.