Reproduce “Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hongjoon Ahn, Heewoong Choi, Jisu Han, Taesup Moon
The paper introduces Option-aware Temporally Abstracted (OTA) value learning, a method to improve value function estimation in hierarchical offline goal-conditioned reinforcement learning for long-horizon tasks. This approach achieves superior performance on benchmarks like OGBench, such as increasing success rates from 12% to 92% on HumanoidMaze-giant by providing more order-consistent value signals to high-level policies.
Key insights
- The primary bottleneck in hierarchical offline GCRL for long-horizon tasks is the high-level policy's failure to generate effective subgoals, not the low-level policy's execution ability.
- This high-level policy failure is rooted in the "order inconsistency" of the value function, which provides unreliable advantage estimates for states far from the goal.
- A specific design of temporal abstraction, where the discount is applied once per option rather than per step, is crucial for learning robust and order-consistent value functions over long horizons.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.