Reproduce “One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learning” on OpenResearch

About this paper

Zijian Guo, İlker Işık, H. M. Sabbir Ahmad, Wenchao Li

The GenZ-LTL framework enables reinforcement learning agents to generalize zero-shot to complex, unseen Linear Temporal Logic (LTL) specifications by adopting a "one subgoal at a time" approach. This method is enhanced with Hamilton-Jacobi reachability for robust safety and a novel observation reduction technique, demonstrating superior success rates and near-zero violations across various tasks and complexities.

Key insights

• Focusing on one immediate reach-avoid subgoal at a time, when combined with explicit safety and smart observation reduction, can lead to superior zero-shot generalization compared to conditioning on complex subgoal sequences.
• Hamilton-Jacobi reachability provides a principled and robust method to enforce safety as a hard constraint within RL, effectively managing conflicts between task progression and safety requirements.
• Reducing observations based on the current subgoal leverages an assumed observational symmetry across propositions, enabling effective scaling and generalization to a larger, unseen set of atomic propositions.

Summary by alphaXiv.