Reproduce “LLM-Based Offline Learning for Embodied Agents via Consistency-Guided Reward Ensemble” on OpenResearch

About this paper

Yujeong Lee, Sangwoo Shin, Wei-Jin Park, Honguk Woo

This paper presents COREN (Consistency-guided Reward ENsemble), a framework that employs Large Language Models to generate dense, spatio-temporally consistent reward feedback during the offline training of embodied agents. This approach facilitates the development of low-latency, resource-efficient agents that achieve performance comparable to state-of-the-art LLM-based agents, while using a significantly smaller policy network and not requiring LLMs at inference time.

Key insights

• Large Language Models can effectively serve as 'reward teachers' during the offline training of embodied agents, eliminating the need for their real-time deployment.
• Ensuring spatio-temporal consistency (contextual, structural, and temporal) in LLM-generated rewards is crucial for accurately grounding them in the environment's domain.
• An ensemble mechanism that learns to combine different LLM-generated reward signals provides a robust and unified dense reward for efficient policy learning.

Summary by alphaXiv.