Reproduce “Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstration” on OpenResearch

About this paper

Rouhollah Rahmatizadeh, Pooya Abolghasemi, Ladislau Bölöni, Sergey Levine

This research presents an end-to-end learning from demonstration framework for inexpensive robots, enabling multi-task vision-based manipulation. It integrates a VAE-GAN for visual feature regularization and an autoregressive multimodal action predictor, achieving success rates between 76% and 88% on five diverse manipulation tasks using only monocular RGB input.

Key insights

• Combining multi-task learning with a shared deep neural network significantly improves performance and generalization across diverse manipulation skills, even on low-cost hardware.
• Auxiliary tasks, such as VAE-GAN-based image reconstruction, effectively regularize visual feature learning, enhancing robustness for fine perception and challenging object properties.
• Utilizing an autoregressive multimodal action predictor is crucial for handling the inherent ambiguity and multiple valid solutions present in human demonstrations, leading to more robust and coherent robot behaviors than unimodal prediction.

Summary by alphaXiv.