Reproduce “AgentHER: Hindsight Experience Replay for LLM Agent Trajectory Relabeling” on OpenResearch

About this paper

Liang Ding

Researchers from The University of Sydney introduced AgentHER, an offline pipeline that recovers training signals from failed LLM agent trajectories by adapting Hindsight Experience Replay to natural-language contexts. This method systematically transforms discarded failures into high-quality training data, achieving performance improvements of up to +11.4% on benchmarks like ToolBench and WebArena while reducing data collection needs by half.

Key insights

• A significant portion of failed LLM agent trajectories can be systematically re-framed into valid successful demonstrations for alternative, implicitly achieved goals, thereby converting data waste into valuable training signals.
• Rigorous cross-model multi-judge verification is crucial for maintaining high label quality and minimizing noise in synthetic hindsight goals, demonstrating superior performance over single-model or same-model consistency checks.
• The method substantially enhances data efficiency and agent performance across various model sizes and complex benchmarks (WebArena, ToolBench) without requiring expensive additional online environment interactions.

Summary by alphaXiv.