Reproduce “SfM-TTR: Using Structure from Motion for Test-Time Refinement of Single-View Depth Networks” on OpenResearch

About this paper

Sergio Izquierdo, Javier Civera

Izquierdo and Civera introduce SfM-TTR, a test-time refinement method that enhances single-view depth prediction by leveraging sparse 3D point clouds from Structure from Motion (SfM) as a self-supervisory signal. This approach consistently reduced common depth error metrics across various baselines, for example, lowering the Abs Rel error for the DIFFNet model from 0.071 to 0.056 on the KITTI dataset, particularly improving accuracy for medium and large depths.

Key insights

• Sparse but geometrically robust wide-baseline information from SfM can serve as a powerful self-supervisory signal for improving dense single-view depth estimation at test time, outperforming photometric TTR.
• Refining only the encoder part of the depth network effectively improves the scene's internal representation while preserving the decoder's ability to generate sharp, dense depth maps.
• A sophisticated, robust scale alignment procedure is essential for effectively integrating sparse, noisy, and potentially unscaled SfM data with deep network predictions.

Summary by alphaXiv.