Reproduce “Sample Efficient Reinforcement Learning via Large Vision Language Model Distillation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Donghoon Lee, Tung M. Luu, Younghwan Lee, Chang D. Yoo
A framework called LVLM2P distills knowledge from large vision-language models (LVLMs) into compact reinforcement learning (RL) agents, enhancing sample efficiency by an average of 2.56 times for PPO and 2.86 times for A2C. This approach enables faster learning for RL agents in decision-making tasks without the computational overhead of deploying large foundation models directly.
Key insights
- The broad, generalized knowledge within LVLMs can be effectively transferred to compact RL agents to accelerate learning, even for tasks the LVLM hasn't been explicitly trained on as a policy.
- Distilling soft probability distributions from the teacher LVLM provides richer information to the student, leading to more robust policies and improved sample efficiency compared to using hard (one-hot) action targets.
- A balanced objective, combining environmental feedback with teacher guidance (controlled by hyperparameter "lambda"), is crucial for the student to leverage teacher knowledge for efficient exploration while adapting to the specific task.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.