Reproduce “One Leaf Reveals the Season: Occlusion-Based Contrastive Learning with Semantic-Aware Views for Efficient Visual Representation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaoyu Yang, Lijian Xu, Hongsheng Li, Shaoting Zhang
Occluded Image Contrastive Learning (OCL) presents a scalable pre-training paradigm for visual representation that generates semantically distinct views through masking and leverages a simplified contrastive learning objective. This approach achieves 85.8% top-1 accuracy on ImageNet-1K fine-tuning for ViT-L/16 with only 133 hours of pre-training on 4 A100 GPUs, demonstrating competitive performance without auxiliary modules or extensive data augmentations.
Key insights
- Generating semantically distinct views by applying global masking and then splitting visible patches into non-overlapping groups effectively addresses conceptual redundancy in contrastive learning.
- High-level semantic features can be learned efficiently from aggregated [CLS] tokens of occluded views, demonstrating that auxiliary MLP heads are not necessary, simplifying the architecture.
- An optimal masking ratio, along with sufficiently large batch sizes, is critical for balancing the information available in views with the diversity needed for effective contrastive learning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.