Reproduce “ICPL: Few-shot In-context Preference Learning via LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chao Yu, Qixin Tan, Hong Lu, Jiaxuan Gao, Xinting Yang, Yu Wang, Yi Wu, Eugene Vinitsky
ICPL is an iterative framework that leverages large language models (LLMs) to learn and refine reward functions for reinforcement learning solely from few-shot in-context human preferences. It achieves a greater than 30-fold reduction in human preference queries compared to traditional preference-based RL methods, demonstrating effective learning and refinement of complex, subjective behaviors such as human-like jumps.
Key insights
- LLMs can effectively perform in-context preference learning, inferring and iteratively refining programmatic reward functions based on abstract human feedback over dynamic behaviors.
- Integrating LLMs into the PbRL loop drastically improves query efficiency, making preference learning practical for tasks previously limited by high human data collection costs.
- The proposed method demonstrates genuine preference learning, as evidenced by monotonic improvement of reward functions over iterations and successful performance with real human feedback.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.