Reproduce “GeoLAM: Learning Geometry-Grounded Latent Actions from Unlabeled Human Videos” on OpenResearch

About this paper

Yifan Xie, Hekun Tian, Jinkun Liu, YuAn Wang, Qiao Sun, Wenbo Ding

GeoLAM learns compact transition representations from human videos by combining future-feature reconstruction with motion targets estimated by a frozen four-dimensional geometry teacher, without hand-pose annotations. It transfers these latent actions to a robot world-action model trained on demonstrations. On LARYBench, GeoLAM reaches 0.29 mean regression MSE and 71.22% classification accuracy versus 0.60 and 45.62% for LAPA-DINOv3, indicating control information, not general improvement.

Key insights

• Explicit 4D geometry supervision resolves part of the ambiguity in future-frame reconstruction, directing the bottleneck toward observable, manipulation-related motion and away from static background changes.
• The representation transfers from human-only video pretraining to robot benchmarks without modeling human hand trajectories directly, supporting cross-embodiment action grounding.
• Geometry-informed reconstruction alone improves representation quality, but adding D4RT supervision provides the largest gain, lowering average LARYBench MSE from 0.46 to 0.29 and raising classification accuracy from 68.76% to 71.22%.

Summary by alphaXiv.