Reproduce “Embodied Referring Expression Comprehension in Human-Robot Interaction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Md Mofijul Islam, Alexi Gladstone, Sujan Sarker, Ganesh Nanduru, Md Fahim, Keyan Du, Aman Chadha, Tariq Iqbal
Researchers from the University of Virginia developed Refer360, a large-scale, multi-view dataset for embodied referring expression comprehension, along with MuRes, a multimodal guided residual module. The MuRes module consistently improved performance on E-RFE tasks and achieved notable gains in Visual Question Answering, for instance, boosting CLIP's VQA accuracy on ScienceQA from 21.31% to 51.85%.
Key insights
- Human communication in object-referencing tasks is overwhelmingly multimodal (96.97% preference), requiring robots to interpret combined verbal commands and nonverbal cues like gaze and pointing.
- Effective multimodal fusion requires selectively reinforcing salient modality-specific information through a guided residual mechanism, rather than uniform integration, to overcome the "representation gap" in pre-trained models.
- The optimal reinforcement strategy for multimodal models is task-dependent, with visual cues being more critical for embodied grounding and balanced cues for abstract reasoning in VQA.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.