Reproduce “Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Junhyuk Oh, Satinder Singh, Honglak Lee, Pushmeet Kohli

A hierarchical reinforcement-learning framework enables agents to execute changing instruction lists by combining reusable parameterized skills, analogy-structured task embeddings, and a meta-controller with a learned decision time-scale. In Minecraft-like and grid-world environments, the method generalized to unseen action-object combinations and longer sequences, reaching up to 99.7% unseen-skill success and 56% success on unseen 20-instruction sequences.

Key insights

  • Structured relationships in task embeddings are essential for compositional zero-shot transfer: removing the analogy objective reduced unseen-task success from 99.5% to 34.8% in one setting and from 99.7% to 2.2% in another.
  • Effective hierarchy requires more than choosing a new skill after termination or at every time step; the controller must learn when to preserve a high-level decision and when to interrupt it.
  • The system separates persistent instruction progress from temporary subtasks, allowing it to exploit newly appearing objects and then return to the interrupted instruction.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.