Reproduce “MatSpray: Fusing 2D Material World Knowledge on 3D Geometry” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Philipp Langsteiner, Jan-Niklas Dihlmann, Hendrik P.A. Lensch
Researchers at the University of Tübingen introduced MatSpray, a framework fusing 2D material priors from diffusion models onto 3D Gaussian Splatting to generate high-quality, relightable 3D assets. This method improves relighting accuracy and material map consistency, particularly for metallic and highly specular objects, while demonstrating faster processing times.
Key insights
- Fusing "world material knowledge" from 2D diffusion models with a 3D Gaussian representation effectively leverages strengths from both domains to provide robust and plausible material priors for 3D reconstruction.
- The Neural Merger, utilizing softmax-based interpolation, effectively resolves inherent multi-view inconsistencies in 2D material predictions, yielding coherent and physically plausible 3D material maps that are disentangled from illumination.
- This integrated approach enables the creation of high-quality relightable assets with accurately disentangled, spatially varying PBR materials, outperforming traditional 3D inverse rendering in quality and efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.