Reproduce “Thresholded Lexicographic Ordered Multiobjective Reinforcement Learning” on OpenResearch

About this paper

Alperen Tercan, Vinayak S. Prabhu

A new policy-gradient approach incorporating the Lexicographic Projection Algorithm (LPA) successfully addresses limitations in existing Thresholded Lexicographic Q-Learning (TLQ) methods for multi-objective reinforcement learning. This method effectively solves problem classes where TLQ previously failed and achieves improved performance in complex multi-objective tasks by ensuring higher-priority objectives are maintained while optimizing lower ones.

Key insights

• Existing TLQ methods fail in a previously undiscussed problem class involving a constrained terminating endpoint objective with an unconstrained non-terminating secondary objective, leading to suboptimal or contradictory agent behavior.
• Policy gradient methods offer a more robust and theoretically sound approach for Thresholded Lexicographic Multi-Objective Reinforcement Learning by directly learning policies rather than relying on value functions that lack Bellman optimality.
• The use of hypercone projections in the LPA provides a parameterized level of 'conservativeness' ($\Delta$), enabling finer control over the trade-off between optimizing lower-priority objectives and strictly preserving higher-priority ones.

Summary by alphaXiv.