Reproduce “DINO Eats CLIP: Adapting Beyond Knowns for Open-set 3D Object Retrieval” on OpenResearch

About this paper

Xinwei He, Yansong Zheng, Qianru Han, Zhichuan Wang, Yuxuan Cai, Yang Zhou, Jingbo Xia, Yulong Wang and 2 more

Researchers from Huazhong Agricultural University and collaborators introduced DINO Eats CLIP (DEC), a framework that adapts DINO vision transformers and employs CLIP for virtual feature synthesis to enhance open-set 3D object retrieval. The DEC framework achieved new state-of-the-art performance across various benchmarks, demonstrating improvements such as a +5.22% mAP gain on OS-MN40-core compared to previous leading methods.

Key insights

• DINO encoders inherently provide robust, fine-grained, and generalizable features for 3D objects, outperforming CLIP in zero-shot 3DOR even without explicit adaptation.
• Hierarchical aggregation of view features through localized chunking and cross-chunk integration within CAM effectively captures inter-view structural dependencies, leading to more discriminative 3D representations.
• Synthesizing virtual features for unseen classes by semantically shifting seen-class features in CLIP's latent space serves as an effective regularization mechanism, significantly enhancing open-set generalization.

Summary by alphaXiv.