Reproduce “ARM: Appearance Reconstruction Model for Relightable 3D Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xiang Feng, Chang Yu, Zoubin Bi, Yintong Shang, Feng Gao, Hongzhi Wu, Kun Zhou, Chenfanfu Jiang and 1 more

The Appearance Reconstruction Model (ARM) provides a framework for generating high-quality 3D meshes and physically-based rendering (PBR) maps, including diffuse albedo, roughness, and metalness, from sparse input images. This system achieves superior geometry reconstruction with an F-Score of 0.968 and appearance quality with a PSNR of 21.692 on the GSO dataset, enabling realistic relighting for generated 3D assets.

Key insights

  • Explicitly decoupling geometry and appearance allows specialized networks to achieve higher fidelity for each task, overcoming the limitations of unified models that struggle with both detailed shape and texture.
  • Processing appearance directly in UV texture space with multi-view measurements and inpainting techniques effectively bypasses the resolution and blurriness constraints inherent in volumetric or triplane representations for textures.
  • Incorporating a material prior significantly enhances the robustness and accuracy of material-illumination decomposition, which is critical for generating correct PBR maps from sparse visual input.
  • Leveraging existing large reconstruction model (LRM) architectures for geometry and glossy properties, while innovating on UV-space texture reconstruction, provides a balanced approach to efficiency and quality.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.