Reproduce “Putting Humans in a Scene: Learning Affordance in 3D Indoor Environments” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xueting Li, Sifei Liu, Kihwan Kim, Xiaolong Wang, Ming-Hsuan Yang, Jan Kautz

Researchers at UC Merced and NVIDIA introduced a method for predicting physically feasible and semantically plausible 3D human poses within 3D indoor environments. This approach involved automatically synthesizing a large-scale dataset of 1.5 million 3D human-scene interactions and training a generative model to produce accurate 3D poses from single RGB, RGB-D, or depth images.

Key insights

  • Automated synthesis of large-scale, physically constrained 3D human-scene interaction data is a viable solution to the dataset scarcity problem, enabling robust data-driven learning for 3D affordance prediction.
  • Decomposing the complex affordance prediction task into separate modules for location ('where') and gesture ('what'), with joint training, significantly enhances both semantic plausibility and physical feasibility of 3D human poses.
  • Incorporating explicit 3D geometric constraints through adversarial training and geometry-specific loss functions is critical for the model to hallucinate and generate physically realistic 3D human poses from potentially ambiguous 2D scene inputs.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.