Reproduce “Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shubham Tulsiani, Tinghui Zhou, Alexei A. Efros, Jitendra Malik

This paper from UC Berkeley introduces Differentiable Ray Consistency (DRC), a method for learning single-view 3D shape prediction from readily available multi-view 2D/2.5D observations instead of costly 3D ground-truth data. DRC achieves performance comparable to 3D-supervised methods on ShapeNet and demonstrates enhanced robustness to noisy training signals, effectively bridging classical geometric consistency with modern deep learning.

Key insights

  • Geometric view consistency principles can be formulated as fully differentiable loss functions, enabling end-to-end training of neural networks for 3D reconstruction from 2D observations.
  • Representing 3D shapes as probabilistic occupancy grids allows for differentiable ray tracing and naturally handles uncertainty, contributing to robustness against noisy supervision.
  • Robust single-view 3D shape prediction can be learned using only multi-view 2D/2.5D data (e.g., masks, depth maps) as supervision, without requiring explicit 3D ground truth.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.