Reproduce “LLM Augmented Hierarchical Agents” on OpenResearch

About this paper

Bharat Prakash, Tim Oates, Tinoosh Mohsenin

Researchers from the University of Maryland, Baltimore County and Johns Hopkins University developed LLM Augmented Hierarchical Reinforcement Learning (LLMxHRL), a framework that uses large language models to guide the high-level exploration of hierarchical agents. This approach improves sample efficiency and task success across various simulated environments and a real-world robot arm, enabling the trained agent to operate independently of the LLM during deployment.

Key insights

• Integrating LLM-derived common sense knowledge as a guiding prior significantly enhances the exploration efficiency and overall task success of hierarchical RL agents.
• Gradually annealing the LLM's influence during training is effective in enabling the agent to internalize learned strategies, ensuring deployment independence from the LLM.
• The framework provides a practical method to bridge high-level symbolic reasoning from LLMs with grounded sub-symbolic learning in RL, reducing the need for manual reward shaping.

Summary by alphaXiv.