Reproduce “OpenClaw-RL: Train Any Agent Simply by Talking” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yinjie Wang, Xuyang Chen, Xiaolong Jin, Mengdi Wang, Ling Yang
OpenClaw-RL updates agents from interaction feedback while they serve requests. Its process reward model scores signals and turns corrections into hints, selected by teacher–student token overlap and combined with reward-based learning. In simulated personalization across three roles, the method reaches the preference criterion in 10.3 sessions, versus 14.1 with GRPO alone; this measures sessions to criterion, not real-user deployment.
Key insights
- Continuous online learning from live, heterogeneous 'next-state signals' is feasible and effective for both personal and general LLM agents.
- Combining evaluative (scalar) and directive (token-level hint) signals in a hybrid RL objective yields greater optimization efficiency and robustness than using either alone.
- Algorithmic mechanisms, such as overlap-guided hint selection and log-probability-difference clipping, are crucial for achieving stable and efficient on-policy distillation in dynamic, online environments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.