Reproduce “LEMON: Learning 3D Human-Object Interaction Relation from 2D Images” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuhang Yang, Wei Zhai, Hongchen Luo, Yang Cao, Zheng-Jun Zha

LEMON predicts human contact, object affordance, and object position from an image, a posed human mesh, and an object point cloud. It combines image-derived intention cues with geometric relationships between the human and object shapes. On the 3DIR test set, its contact geodesic error is 7.55 cm versus 15.25 cm for DECO, under the reported benchmark setup.

Key insights

  • Jointly modeling all 3D human-object interaction elements (contact, affordance, spatial relation) leads to a more comprehensive and accurate understanding than isolated predictions.
  • Explicitly capturing "intention affinity" (how object functionality influences human intent) and "geometry affinity" (how human and object shapes align) is crucial for resolving interaction ambiguities and inferring hidden details.
  • Leveraging human contact as a direct constraint significantly improves the prediction of the 3D spatial relationship between the human and the object.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.