Reproduce “SCALAR: Learning and Composing Skills through LLM Guided Symbolic Planning and Deep RL Grounding” on OpenResearch

About this paper

Renos Zabounidis, Yue Wu, Simon Stepputtis, Woojun Kim, Yuanzhi Li, Tom Mitchell, Katia Sycara

The SCALAR framework enables large language models to acquire low-level control skills for complex environments by integrating LLM-guided symbolic planning with deep reinforcement learning through a self-supervised, iterative feedback loop. This system achieved an 88.2% diamond collection rate in Craftax-Classic, a 1.9x improvement over strong baselines, and successfully completed a challenging 'Enter Gnomish Mines' task where prior methods failed.

Key insights

• Bidirectional feedback between LLM planning and RL execution enables iterative refinement of skill specifications, correcting initial LLM hypotheses and improving robustness to environmental changes.
• Decomposing complex tasks into composable "skills" with focused reward functions makes long-horizon, sparse-reward RL tasks tractable.
• Efficient training strategies, like Frontier Checkpointing, are essential for deep prerequisite chains, allowing more training frames to be dedicated to the target skill.

Summary by alphaXiv.