Reproduce “Hand-in-the-Loop: Improving VLA Policies for Dexterous Manipulation via Seamless Hand-Arm Intervention” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhuohang Li, Liqun Huang, Wei Xu, Zhengming Zhu, Nie Lin, Xiao Ma, Xinjun Sheng, Ruoshi Wen

HandITL is a human-in-the-loop intervention system designed to refine Vision-Language-Action (VLA) policies for dexterous manipulation through seamless hand-arm intervention. It achieves a 99.8% reduction in command discontinuities during human takeover, leading to VLA policies that improve average sub-goal completion rates by 19% on complex, long-horizon tasks.

Key insights

  • Seamless and continuous human intervention, free from command discontinuities, is crucial for effectively correcting and maintaining stability in high-DoF dexterous manipulation tasks.
  • On-policy correction data, collected specifically from policy-induced failure states, is more effective for improving the robustness of VLA policies on long-horizon tasks than simply adding more off-policy teleoperation demonstrations.
  • A shared-control (Copilot) intervention mode, which blends local human residual corrections with the policy's primary control, generates intervention data closer to the policy's rollout distribution, leading to more stable and effective policy refinements.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.