Reproduce “Scene and Human in One World: Reconstruction in a Feedforward Pass” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Boao Shi, Qiao Feng, Yiming Huang, Lingjie Liu
SHOW reconstructs humans and surrounding scenes from moving monocular video by jointly using human cues to guide scene geometry and scene geometry to estimate body scale and placement. On EMDB-1, HS-V is 0.013 versus UniSH’s 0.033, but Human3R has better HS-CF5, 1.13 versus SHOW’s 5.88. Trained on synthetic BEDLAM2.0, it remains difficult for very small people and extreme scale variations.
Key insights
- Deeply coupling human and scene reconstruction within a unified feed-forward pipeline enables more consistent metric-scale 3D representations than decoupled or weakly coupled approaches.
- Integrating human-specific cues, such as DensePose supervision and mask prompting, into a geometry foundation model significantly improves human-aware scene understanding.
- Explicitly enforcing scale and placement consistency through joint optimization and novel alignment losses is crucial for achieving physically plausible human-scene interactions from monocular input.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.