Reproduce “Locality-Aware Zero-Shot Human-Object Interaction Detection” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sanghyun Kim, Deunsol Jung, Minsu Cho

Researchers from POSTECH developed the Locality-Aware Interaction Network (LAIN) to infuse CLIP's global representations with fine-grained locality and interaction awareness, specifically for zero-shot Human-Object Interaction detection. This approach consistently improved performance on the HICO-DET benchmark, achieving 31.83 mAP in the RF-UC setting and showing notable gains on rare and small object interactions with minimal computational overhead.

Key insights

  • Enhancing global VLM representations with specific mechanisms for fine-grained locality and relational interaction awareness effectively bridges the domain gap for complex region-level tasks like HOI detection.
  • Incorporating spatial context and neighborhood information into patch tokens improves the model's ability to capture subtle visual cues for individual objects.
  • Explicitly modeling interaction patterns between human and object regions through contextual reasoning is essential for discerning the nature and presence of interactions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.