Reproduce “A Visual Reinforcement Learning-Based Separate Primitive Policy for Peg-in-Hole Tasks” on OpenResearch

About this paper

Zichun Xu, Zhaomin Wang, Yuntao Li, Lei Zhuang, Zhiyuan Zhao, Guocai Yang, Jingdong Zhao

A task-specific visual reinforcement learning design separates peg-in-hole assembly into sequential localization and insertion policies rather than learning a single joint controller or a high-level primitive selector. Built on DrQ-v2, the approach improves simulated sample efficiency, remains effective under force penalties, and transfers zero-shot to hardware with 199 successes in 200 real-world trials across five hole geometries.

Key insights

• A manually specified behavioral structure can reduce visual RL exploration demands when the task naturally alternates between alignment and insertion; a learned high-level selector is not always necessary.
• Restricting the insertion primitive to the contact-sensitive axis gives it clearer responsibility for adapting to changing contact conditions and helps preserve performance under a force penalty above 15 N.
• The benefit comes from primitive-level feedback and action decomposition rather than simply repeating actions: adding action repeat to the baseline or to S2P did not reproduce the improvement.

Summary by alphaXiv.