Reproduce “Dyna-DINO: Efficient ViT Distillation Via Adaptive Representation Anchoring” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiaqi Zhang, Ashton Lee, Anthony Wong, John Zou, Sami BuGhanem, Randall Balestriero
Dyna-DINO distills a large vision transformer into a smaller student by moving its feature target from shallow to deeper teacher layers as online representation similarity increases. On ImageNet-100, a ViT-Small student trained from a ViT-G teacher for 100 epochs reaches 90.10% patch-token linear-probe accuracy, versus 77.86% for a fixed final-layer target; this result is specific to that setup.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.