Reproduce “CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learning” on OpenResearch

About this paper

Zhao Mandi, Homanga Bharadhwaj, Vincent Moens, Shuran Song, Aravind Rajeswaran, Vikash Kumar

Researchers introduce CACTI, a framework designed for scalable multi-task and multi-scene visual imitation learning in robotics, which efficiently generates diverse training data using semantic augmentation and pre-trained visual representations. The approach enables policies to achieve 30% success on 10 real-world manipulation tasks and 62% success across 18 tasks in 100 simulated kitchen layouts, demonstrating generalization to new environments.

Key insights

• CACTI's modular design and data-centric approach offer a practical and efficient alternative to end-to-end Reinforcement Learning for scaling robot learning, which often fails in complex multi-task, multi-scene settings.
• Semantic data augmentations, particularly through generative in-painting, are critical for significantly improving policy robustness and generalization to novel objects and diverse scene layouts.
• Leveraging pre-trained visual representations from large internet-scale datasets effectively transfers knowledge to robotics, making these representations highly competitive even when frozen or used out-of-domain for robotic tasks.

Summary by alphaXiv.