Reproduce “Your Transformer Is Secretly Elastic” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Walter Simoncini, Dheeraj Varghese, Gertjan J. Burghouts, Cees G.M. Snoek
The method lets one pretrained encoder serve different compute budgets by ranking its MLP neurons and attention heads, then removing lower-ranked units without retraining or updating model weights. At 50% FLOP sparsity, DINOv3 ViT-H+/16 retains 84.4% ImageNet-1k linear-probing accuracy, down 0.8 percentage points, while removing 427 million parameters. Deep pruning remains difficult for evaluated decoder-based text embedders.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.