Reproduce “UniRefiner: Teaching Pre-trained ViTs to Self-Dispose Dross via Contrastive Register” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Congpei Qiu, Zhaoyu Hu, Wei Ke, Zhuotao Tian, Yanhao Wu, Tong Zhang
UniRefiner is a self-supervised framework designed to refine pre-trained Vision Transformers by teaching them to identify and absorb 'spurious tokens' into a dedicated register. This method consistently enhances spatial representations, leading to performance improvements of up to 9.4% mIoU on ADE20K for semantic segmentation and up to +19% mIoU in zero-shot Cityscapes for vision-language tasks.
Key insights
- Spurious tokens in large ViTs are not monolithic but comprise diverse types (FP, GP, AH) that require specific identification and handling, accounting for over 40% of features in some models.
- A dedicated 'register' mechanism, formed by injecting noise patches, can effectively serve as an architectural sink to isolate and redistribute spurious information away from core image representations.
- Contrastive distillation, guided by a robust spurious token filter derived from a frozen teacher, allows for lightweight fine-tuning to purify spatial features while maintaining the pre-trained model's multimodal alignment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.