Reproduce “Preference-Calibrated Human-in-the-Loop Reinforcement Learning for Robotic Manipulation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zeyi Liu, Guangyao Liu, Yinuo Qu, Yuquan Xue, Bofang Jia, Chunhua Yang, Weihua Gui, Keke Huang and 1 more
A human-in-the-loop reinforcement learning framework named PACT enhances robotic manipulation by using implicit human preference signals to calibrate Q-value updates and actor optimization, which leads to superior success rates and faster training on real-robot tasks. The method improved average success rate by 24.5% and reduced intervention rate by 14.8% across five complex manipulation tasks compared to existing approaches.
Key insights
- Successful human-in-the-loop trajectories are inherently heterogeneous, consisting of normal, suboptimal, and human-intervention segments, making uniform credit assignment suboptimal.
- Implicit preference signals, naturally occurring at the point of human intervention, can be effectively leveraged for fine-grained, segment-level Q-value correction and direct policy guidance.
- Accurately identifying and correcting Q-value overestimation in suboptimal segments leads to more stable critic learning and faster, more efficient policy convergence.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.