Reproduce “YOLOA: Real-Time Affordance Detection via LLM Adapter” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuqi Ji, Junjie Ke, Lihuo He, Jun Liu, Kaifan Zhang, Yu-Kun Lai, Guiguang Ding, Xinbo Gao
YOLOA introduces a real-time system that unifies object detection and affordance learning, leveraging an LLM Adapter for language-guided cross-modal refinement. This approach achieved a mAP of 52.8 on ADG-Det and 73.1 on IIT-Heat, while its lightweight variant reached up to 846.24 FPS.
Key insights
- Effectively integrating semantic knowledge from LLMs via an adapter can mutually enhance visual object detection and affordance learning in a closed-loop fashion.
- Distilling LLM knowledge during training allows for significant speedups during inference by removing the LLM component, achieving real-time performance without sacrificing much accuracy.
- Simultaneously addressing the 'what-where-how' aspects of object understanding provides a more comprehensive and actionable perception for embodied AI systems.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.