Reproduce “RayTun3R: Online Camera Adaptation in 3D Foundation Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Daniil Sinitsyn, Nikita Araslanov, Daniel Cremers

A method called RayTun3R facilitates accurate depth and camera pose recovery from fisheye images using 3D foundation models originally trained on pinhole data, incurring negligible inference overhead. It consistently reduces translation-direction error by over 3x and rotation error by over 12x on datasets like ETH3D, outperforming other adaptation techniques.

Key insights

  • The degradation of 3D foundation models on fisheye imagery stems from a fundamental pinhole camera bias inherent in their positional encodings.
  • Adapting camera-dependent positional components, rather than generic feature transformations, provides a more effective and efficient strategy for addressing camera-geometry-induced domain shifts.
  • Combining learned radial and angular residuals for positional embeddings with parameter-free geometric corrections for patches and prediction grids yields optimal performance in correcting fisheye distortion.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.