Reproduce “Language as an Abstraction for Hierarchical Deep Reinforcement Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yiding Jiang, Shixiang Gu, Kevin Murphy, Chelsea Finn

A hierarchical deep reinforcement learning framework called HAL integrates compositional language as an abstraction, enabling agents to solve complex, long-horizon manipulation tasks more efficiently. The approach, which utilizes Hindsight Instruction Relabeling for robust low-level policy training, demonstrated superior performance over existing baselines and systematic generalization to novel instructions, even from raw pixel observations.

Key insights

  • The explicit compositional structure of language is critical for effective learning and systematic generalization in complex tasks, outperforming non-compositional latent representations.
  • Hindsight Instruction Relabeling (HIR) is essential for efficiently training language-conditioned policies by transforming sparse binary rewards into a denser, more effective learning signal.
  • Acting in the abstract space of language instructions provides a form of structured exploration, leading to faster learning and higher rewards during initial training phases compared to non-hierarchical methods.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.