Reproduce “Affogato: Open-Vocabulary Affordance Grounding with Automated Data Generation at Scale” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Junha Lee, Eunha Park, Chunghyun Park, Dahyun Kang, Minsu Cho

Researchers at Pohang University of Science and Engineering (POSTECH) developed Affogato, an automated framework that generates Affogato-750K, a large-scale, open-vocabulary dataset for 3D point cloud and 2D image affordance grounding. Pretraining on this dataset consistently improved model performance across modalities, enhancing generalization to unseen object/affordance categories and demonstrating zero-shot transferability to real-world robotics scenes.

Key insights

  • Existing foundation models can be effectively combined to automatically generate large-scale, high-quality, open-vocabulary data for complex visual tasks like affordance grounding, overcoming manual annotation bottlenecks.
  • The generated Affogato-750K dataset provides broadly transferable supervision across 2D and 3D modalities, significantly enhancing model generalization to novel object and affordance categories, as well as real-world scenarios.
  • Careful design elements within the data generation pipeline, such as CoT prompting for queries and multi-view consensus for heatmaps, are crucial for ensuring the functional grounding and precise granularity of automatically created annotations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.