Reproduce “Preference-Based Long-Horizon Robotic Stacking with Multimodal Large Language Models” on OpenResearch

About this paper

Wanming Yu, Adrian Röfer, Abhinav Valada, Sethu Vijayakumar

Researchers from the University of Edinburgh and Freiburg developed a framework enabling multimodal Large Language Models to plan long-horizon robotic stacking tasks, considering both visible and hidden object properties like weight and internal stability, alongside human preferences. The fine-tuned LLMs achieved substantially higher success rates in simulation for tasks involving latent properties and demonstrated dynamic replanning on a real robot, including unstacking and restacking actions.

Key insights

• Fine-tuning LLMs with a physics-simulation-generated dataset vastly improves their success rates in long-horizon stacking, particularly for tasks involving hidden object properties, over prompt-tuned models.
• Multimodal sensing, specifically force-torque for weight and audio for internal stability, effectively enables LLMs to incorporate non-visual physical properties into their reasoning for preference-based robotic manipulation.
• LLM-driven planning supports dynamic, online adaptation on a real robot, allowing for plan adjustments and complex actions like unstacking and restacking to meet evolving preferences or newly acquired object information.

Summary by alphaXiv.