Reproduce “Mental Modeling of Reinforcement Learning Agents by Language Models” on OpenResearch

About this paper

Wenhao Lu, Xufeng Zhao, Josua Spisak, Jae Hee Lee, Stefan Wermter

An empirical study investigates the capability of Large Language Models (LLMs) to construct "mental models" of reinforcement learning (RL) agents by inferring agent policies and environmental dynamics from interaction history. Findings indicate LLMs can partially model simpler tasks, such as predicting agent actions in MountainCar with 86.18% accuracy, but performance degrades with increasing task complexity and requires careful context provisioning, with explicit dynamics principles significantly improving understanding.

Key insights

• Large Language Models can establish partial mental models for simpler reinforcement learning tasks by leveraging common-sense knowledge and inferring from interaction history.
• The processing of interaction history for in-context learning is non-linear, with benefits saturating and sometimes degrading with excessive length, and LLMs surprisingly predict absolute continuous actions better than binned categories.
• Providing explicit "dynamics principles" (e.g., equations) in the system prompt significantly enhances LLMs' ability to understand both agent behavior and environmental transitions, indicating their capacity to integrate structured physical knowledge.

Summary by alphaXiv.