Reproduce “Interaction-aware Representation Modeling with Co-occurrence Consistency for Egocentric Hand-Object Parsing” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuejiao Su, Yi Wang, Lei Yao, Yawen Cui, Lap-Pui Chau

Researchers from The Hong Kong Polytechnic University developed InterFormer, an end-to-end framework for egocentric hand-object segmentation that incorporates dynamic query generation, dual-context feature selection, and a conditional co-occurrence loss. It achieved state-of-the-art mean IoU of 73.22% on the EgoHOS in-domain test set and robust generalization on out-of-domain datasets, improving physical consistency in predictions.

Key insights

  • Explicitly localizing hand-object interaction boundaries provides crucial spatial constraints that guide the network's focus to relevant areas.
  • Dynamically generated queries, grounded in active interaction regions, are more robust and adaptive than static or generically sampled queries, improving initial representation.
  • Combining semantic and boundary-guided features through a dual-context selection process allows for more precise and interaction-relevant feature refinement.
  • Enforcing physical co-occurrence constraints based on hand presence directly mitigates implausible segmentation outputs, making predictions more reliable for real-world applications.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.