Reproduce “HOI-Ref: Hand-Object Interaction Referral in Egocentric Vision” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Siddhant Bansal, Michael Wray, Dima Damen
A new task, HOI-Ref, is introduced for understanding hand-object interactions in egocentric vision, along with HOI-QA, a 3.9 million QA pair dataset. The VLM4HOI model, specialized for this task, outperforms a general-purpose VLM baseline like MiniGPT-v2 in identifying and spatially localizing hand-object interactions.
Key insights
- Specialized training on egocentric data significantly improves VLM performance for hand-object interaction understanding compared to models trained on third-person images.
- Incorporating explicit spatial information (bounding boxes) into the VLM architecture is crucial for accurately localizing interacting hands and objects in egocentric views.
- Large-scale, diverse egocentric datasets (like HOI-QA combining EPIC-Kitchens and Ego4D) are essential for training robust models capable of generalizing across various interaction types and environments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.