Reproduce “Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge, Xiangyu Zhang, Kaisheng Ma, Li Yi

The Contrast with Reconstruct (RECON) framework unifies generative masked modeling and contrastive learning to enhance 3D self-supervised representation learning. This approach addresses data scarcity in 3D vision and achieves new state-of-the-art results across various downstream tasks, notably improving accuracy on ScanObjectNN PB T50 RS by 5.6% over previous methods.

Key insights

  • Generative reconstruction can effectively guide contrastive learning, preventing representation over-fitting and improving generalization in scenarios with limited 3D data.
  • The 'stop-gradient' mechanism in the RECON-block's cross-attention is crucial for decoupling local and global learning signals, resolving task conflicts between generative and contrastive objectives.
  • Leveraging strong, frozen multimodal teachers (2D images, text) provides valuable, diverse 'dark knowledge' for 3D representation learning, aiding in the development of capable 3D models from multimodal data.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.