Reproduce “Real-time Holistic Robot Pose Estimation with Unknown States” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shikun Ban, Juling Fan, Xiaoxuan Ma, Wentao Zhu, Yu Qiao, Yizhou Wang

This paper presents an end-to-end, single feed-forward neural network framework for real-time holistic robot pose estimation from monocular RGB images, even when internal joint states are unknown. The proposed method achieves a 12x speed increase over prior state-of-the-art while maintaining high accuracy on benchmark datasets like DREAM.

Key insights

  • Factorizing the complex holistic pose problem into manageable sub-tasks for modular, feed-forward networks avoids the computational burden of iterative optimization or PnP methods.
  • Specialized modules like DepthNet disentangle scale-variant depth from other scale-invariant features, and KeypointNet effectively bridges pixel-aligned features with the robotic parameter space.
  • Self-supervised consistency regularizations (keypoint and mask consistency) enable effective sim-to-real transfer and improve generalization to real-world images without explicit ground-truth labels.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.