Reproduce “MRHER: Model-based Relay Hindsight Experience Replay for Sequential Object Manipulation Tasks with Sparse Rewards” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuming Huang, Bin Ren, Ziming Xu, Lianghong Wu

MRHER presents a reinforcement learning framework, combining task decomposition and a novel Foresight Relabeling method, to improve sample efficiency in sequential object manipulation tasks with sparse rewards. The framework achieves higher success rates and reduced interaction steps compared to existing baselines on robotic manipulation benchmarks.

Key insights

  • Decomposing complex tasks into stages and using a self-guided exploration strategy, which mixes learned and learning policies, accelerates the acquisition of skills for sequential manipulation.
  • Training dynamics models to predict state *differences* rather than absolute next states improves their ability to capture fine-grained action effects, particularly when objects are manipulated.
  • Foresight Relabeling (FR) mitigates the Identical Non-Negative Reward problem by intelligently sampling diverse starting states from historical trajectories and generating goal-oriented virtual future trajectories using the newest policy.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.