Reproduce “Meshtryoshka: Differentiable Rendering of Real-World Scenes via Mesh Rasterization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
David Charatan, Daniel Xu, Richard Szeliski, George Kopanas, Vincent Sitzmann
Meshtryoshka reconstructs unbounded real-world scenes for novel-view rendering while optimizing a mesh representation directly. It extracts nested triangle shells from a learned field, renders them with a conventional rasterizer, and sends image-loss gradients through interpolation and compositing rather than vertex positions. On Mip-NeRF 360, it reports 24.34 PSNR, below 27.21 for 3D Gaussian Splatting and 28.54 for Zip-NeRF.
Key insights
- Multiple nested shells provide a surface-aware rendering signal while allowing the rasterizer to process only the first visible triangle per shell; the nearly opaque innermost shell makes later intersections largely unnecessary.
- The final zero-level mesh achieves nearly the same NeRF-Synthetic quality as the full multilayer renderer, suggesting that the additional shells mainly stabilize optimization rather than serve as the required deployment representation.
- The approach demonstrates that differentiable attribute interpolation can substitute for direct differentiation through mesh vertex motion, preserving compatibility with standard triangle-rendering pipelines.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.