Reproduce “ForceSight: Text-Guided Mobile Manipulation with Visual-Force Goals” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jeremy A. Collins, Cody Houff, You Liang Tan, Charles C. Kemp

ForceSight, developed at the Georgia Institute of Technology, enables text-guided mobile manipulation by predicting human-interpretable visual-force goals directly from vision and language inputs. The system achieved an 81% task success rate in unseen environments with unseen objects, demonstrating that explicitly incorporating target forces into planning significantly improves performance in contact-rich tasks compared to vision-only approaches.

Key insights

  • Explicitly predicting and utilizing 'visual-force goals' (desired kinematic poses and contact forces) is crucial for robust mobile manipulation, particularly in contact-rich tasks, leading to a substantial increase in task success compared to vision-only approaches.
  • Large pre-trained Vision Transformers, combined with text conditioning, enable effective generalization to unseen environments and object instances with a modest amount of training data.
  • Formulating goals relative to an eye-in-hand camera frame enhances the system's adaptability and robustness, making it suitable for versatile mobile manipulation platforms.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.