Reproduce “Learning to Learn Faster from Human Feedback with Language Model Predictive Control” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jacky Liang, Fei Xia, Wenhao Yu, Andy Zeng, Montserrat Gonzalez Arenas, Maria Attarian, Maria Bauza, Matthew Bennice and 42 more
LMPC fine-tunes PaLM 2 on successful robot-teaching conversations and searches predicted future interactions to choose each next code response. Unlike predicting only final code, LMPC-Rollouts models intermediate feedback and replans after each human turn. On unseen test tasks across five robot embodiments, it achieved 66.3% session success versus 39.4% for PaLM 2-S; most evaluation and training used simulation.
Key insights
- Combining online fast adaptation (in-context learning) with offline slow adaptation (model fine-tuning) overcomes the limitations of LLM context windows, enabling robots to learn from multi-turn human feedback.
- Fine-tuning the LLM on successful human-robot interaction trajectories allows it to learn the 'dynamics' of efficient teaching interactions, predicting future steps to success.
- Conditioning the LLM on 'top-user' data during inference, even for all users, effectively leverages high-quality teaching styles to improve overall teachability and task success.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.