Reproduce “Bring My Cup! Personalizing Vision-Language-Action Models with Visual Attentive Prompting” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sangoh Lee, Sangwoo Mo, Wook-Shin Han
This research from POSTECH introduces Visual Attentive Prompting (VAP), a training-free framework that enables frozen Vision-Language-Action (VLA) models to reliably identify and manipulate specific, user-defined objects among visually similar distractors using a few reference images. The method significantly improved task success rates for personalized pick-and-place and selection tasks, achieving an 80.0% success rate on real-world selection tasks and boosting pick-and-place success to 58.8% for novel objects.
Key insights
- Combined visual highlighting and instruction rewriting are crucial for effective instance-level disambiguation; neither visual prompts nor language rewrites alone are sufficient.
- Spatio-temporal mask propagation is essential for embodied manipulation, ensuring consistent object identity across continuous execution and different camera views, which static visual prompts or learned token embeddings cannot achieve.
- Leveraging off-the-shelf vision foundation models (DINOv2, Grounding DINO, SAM2) provides a robust and computationally efficient perception module that enables training-free adaptation for VLA personalization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.