Reproduce “CLEANER: Self-Purified Trajectories Boost Agentic Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianshi Xu, Yuteng Chen, Meng Li
CLEANER is a framework employing Similarity-Aware Adaptive Rollback (SAAR) to purify training trajectories for agentic Large Language Models, retrospectively correcting execution failures. This method yields more efficient Reinforcement Learning and boosts performance for parameter-constrained LLMs on complex reasoning and coding tasks.
Key insights
- Leveraging a model's intrinsic self-correction capabilities to purify trajectories during data collection effectively mitigates execution noise and improves agentic RL performance.
- Distinguishing between shallow (implementation-level) and deep (reasoning-level) errors allows for targeted adaptive replacement strategies, enhancing learning signal distillation.
- Training on self-purified trajectories enables LLMs to internalize correct reasoning patterns and robust tool usage, reducing reliance on external debugging during inference.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.