Reproduce “GenAug: Retargeting behaviors to unseen situations via Generative Augmentation” on OpenResearch

About this paper

Zoey Chen, Sho Kiami, Abhishek Gupta, Vikash Kumar

A framework leveraging depth-guided text-to-image generative models produces semantically diverse and physically plausible training data for robot manipulation tasks. This approach, called GenAug, substantially improves the generalization capabilities of robot policies, achieving an average absolute improvement of 40% in success rates for unseen real-world environments and objects compared to policies without this augmentation.

Key insights

• Large-scale pre-trained image-text generative models can serve as powerful priors for robot learning, enabling semantic data augmentation that goes beyond simple pixel-level perturbations.
• Generating physically plausible and action-invariant augmented scenes is critical for effective generalization, especially when dealing with unseen objects and complex environment changes.
• The integration of depth information and 3D object meshes with generative models facilitates the creation of geometrically consistent and visually realistic new objects and scenes.

Summary by alphaXiv.