Reproduce “PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithm” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Toygun Basaklar, Suat Gumussoy, Umit Y. Ogras

PD-MORL develops an algorithm that trains a single universal network to generate Pareto-optimal policies for any given preference vector at run-time. It yields broader and denser Pareto fronts with an order of magnitude fewer parameters across discrete and continuous control tasks compared to multi-policy baselines, addressing the challenge of dynamic adaptation and scalability in multi-objective reinforcement learning.

Key insights

  • A single neural network can learn a comprehensive set of Pareto-optimal behaviors, enabling run-time adaptation to varying preferences without retraining or storing multiple policies.
  • The novel preference-driven optimality operator, incorporating a cosine similarity term and interpolation, effectively aligns learned Q-values with desired preferences, especially across objectives with widely varying magnitudes.
  • Combining Hindsight Experience Replay (HER) with parallel exploration significantly enhances sample efficiency and ensures comprehensive coverage of the multi-objective preference space during training.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.