Reproduce “A Reward-Free Viewpoint on Multi-Objective Reinforcement Learning” on OpenResearch

About this paper

Ying-Tu Chen, Wei Hung, Bing-Shu Wu, Zhang-Wei Hong, Ping-Chun Hsieh

Researchers from National Yang Ming Chiao Tung University and MIT integrated Reward-Free Reinforcement Learning (RFRL) into Multi-Objective Reinforcement Learning (MORL) by adapting the Forward-Backward algorithm into a new framework called MORL-FB. This approach achieved superior performance and data efficiency across various continuous and discrete control tasks, showcasing robust generalization to unseen preferences and zero-shot transfer capabilities across tasks with different objective counts.

Key insights

• RFRL's objective to learn optimal policies for any reward function can serve as an effective auxiliary task for MORL, promoting broader knowledge sharing and generalization beyond predefined scalarizations.
• Effective integration of RFRL into MORL requires specific algorithmic enhancements, such as guided exploration and direct utilization of observed multi-objective rewards, to overcome the limitations of naive RFRL application.
• Disentangled representations of environmental dynamics, learned through the adapted FB framework, enable robust generalization to unseen preferences and facilitate zero-shot transfer across tasks with varying numbers of objectives.

Summary by alphaXiv.