Reproduce “GIF: Agentic Generation of Interactive and Functional Object Compositions for Robot Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Long Xu, Zhiqi Zhang, Mi Yan, Shengliang Deng, Chong Xia, Mingyu Dong, Jiayi Chen, Jiangran Lyu and 3 more
GIF generates task-ready compositions of rigid object pairs from language or image prompts. Its pipeline reconstructs objects separately, refines their relative poses, and checks whether the intended contact survives simulation. On a 120-scene benchmark, it averages 7.82/10 in language-prompt Relationship Matching versus 6.33 for SceneSmith. Its language-prompt collision rate is 0.008; real-robot tests cover only two tasks.
Key insights
- Disentangled 2D-3D generative asset construction (CoGen) effectively mitigates issues of object fusion and incompleteness caused by occlusion in contact-rich scenes, providing better initial meshes and poses.
- Combining geometry-guided and physics-guided pose refinement, including black-box simulation feedback, is crucial for achieving physically stable and functionally precise object compositions.
- VLM-based verification and selection at multiple stages ensure high-quality, simulation-ready scenes that accurately reflect target relations and physical stability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.