Reproduce “Scalable Patch-Level Self-Supervised Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Maximilian Seitzer, Gabriele Trivigno, Antonín Vobecký, Seungeun Yi, Maxime Oquab, Huy V. Vo, Oriane Siméoni, Piotr Bojanowski
JEM trains representations by having a student predict a teacher’s descriptions of corresponding image patches across views, with safeguards against information loss and spatial collapse. Its patch-only approach reaches a 23.9 panoptic segmentation score on COCONut at 7B parameters, versus 21.6 for DINOv3 at 7B. Panoptic segmentation assigns categories to pixels while distinguishing countable objects. The comparison is limited by unequal training data, duration, and refinement.
Key insights
- Alignment and the global regularizer are central: removing the latter causes severe collapse, including a drop in ADE20k mIoU from 48.1 to 0.5 in a ViT-L ablation.
- The structural regularizer has a modest effect at ViT-L scale but becomes more important at ViT-g scale, where disabling it reduces ADE20k mIoU by 2.7 points.
- Independent photometric augmentations and alignment of visible patches both contribute to dense-task performance, indicating that spatially localized learning signals alone are not sufficient.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.