Reproduce “ExploRLLM: Guiding Exploration in Reinforcement Learning with Large Language Models” on OpenResearch

About this paper

Runyu Ma, Jelle Luijkx, Zlatan Ajanovic, Jens Kober

The ExploRLLM framework integrates Large Language Models (LLMs) and Vision-Language Models (VLMs) into a Reinforcement Learning (RL) agent to guide exploration in robot manipulation tasks. It employs LLMs to generate hierarchical exploratory code policies and VLMs to refine observation and action spaces, leading to improved RL convergence, higher success rates, and effective zero-shot sim-to-real transfer on a Franka Panda robot.

Key insights

• Integrating LLM-generated, hierarchical code policies as an exploration strategy significantly accelerates RL agent convergence and improves sample efficiency in complex robotic tasks.
• Refining observation and action spaces using FMs (VLMs for object-centric features, residual action formulation) allows the RL agent to learn more effectively and precisely.
• Leveraging the strengths of FMs for high-level reasoning and RL for grounded interaction enables compensation for FM limitations like unreliability and lack of physical understanding.

Summary by alphaXiv.