Reproduce “Template-Free Single-View 3D Human Digitalization with Diffusion-Guided LRM” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhenzhen Weng, Jingyuan Liu, Hao Tan, Zhan Xu, Yang Zhou, Serena Yeung-Levy, Jimei Yang
Human-LRM introduces a template-free, three-stage method for reconstructing high-fidelity 3D human geometry and appearance from a single image. It leverages a generalizable NeRF approach guided by a conditional diffusion model to enhance details and plausibly hallucinate occluded regions, achieving state-of-the-art performance across geometry and appearance metrics with robust generalization.
Key insights
- Combining a generalizable NeRF (LRM) with a conditional diffusion model in a coarse-to-fine manner effectively addresses the challenge of high-fidelity 3D human digitalization, particularly for fine details and occluded areas.
- Diffusion models can be effectively conditioned on coarse 3D priors (rendered RGB, depth, weights sum from initial NeRF) and global image features to generate multi-view consistent, photo-realistic novel images for 3D guidance.
- Employing Signed Distance Functions (SDF) and supervising with normal and depth maps during NeRF training significantly improves the accuracy and detail of the reconstructed 3D geometry.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.