Reproduce “Multiview Equivariance Improves 3D Correspondence Understanding with Minimal Feature Finetuning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yang You, Yixin Li, Congyue Deng, Yue Wang, Leonidas Guibas

Researchers from Stanford University and USC developed a lightweight finetuning strategy to improve the 3D correspondence understanding of existing 2D vision models by leveraging multiview equivariance. This method, which uses SmoothAP loss and a single convolutional layer, achieved performance increases across tasks like object pose estimation (e.g., DINOv2 improved to 9.58 3cm-3deg accuracy), video tracking, and semantic correspondence, while requiring minimal data and computation.

Key insights

  • Large 2D vision models possess an inherent degree of 3D equivariance, with DINOv2 demonstrating the strongest natural capabilities among tested models in generating view-consistent features.
  • There is a strong positive correlation between the quality of 3D equivariant features (measured by Average Pixel Error) and performance on downstream 3D tasks such as object pose estimation, video tracking, and semantic correspondence.
  • Significant improvements in 3D correspondence understanding and downstream task performance can be achieved with extremely few-shot finetuning, even using a single multi-view pair from one object for a single iteration.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.