Reproduce “CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flow” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Philippe Weinzaepfel, Thomas Lucas, Vincent Leroy, Yohann Cabon, Vaibhav Arora, Romain Brégier, Gabriela Csurka, Leonid Antsfeld and 2 more

This work introduces CroCo v2, an enhanced self-supervised pre-training method that enables a generic Vision Transformer architecture to achieve state-of-the-art performance on dense geometric vision tasks like stereo matching and optical flow. It leverages millions of real-world image pairs, employs relative positional embeddings, and scales up the model, demonstrating that specialized architectural components are not strictly necessary for these tasks.

Key insights

  • A generic Vision Transformer-based architecture, when pre-trained with an appropriate cross-view completion task on large-scale real-world data, can achieve state-of-the-art performance on dense geometric tasks without specialized task-specific components.
  • Utilizing relative positional embeddings (RoPE) and scaling up the model's encoder and decoder significantly improves performance and generalization for pixel-level geometric tasks.
  • Learning a probabilistic Laplacian loss provides uncertainty estimates that correlate well with prediction errors, leading to more robust training and inference.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.