Reproduce “Green Hierarchical Vision Transformer for Masked Image Modeling” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Lang Huang, Shan You, Mingkai Zheng, Fei Wang, Chen Qian, Toshihiko Yamasaki

Researchers from University of Tokyo, SenseTime, and University of Sydney developed Green Hierarchical Vision Transformer (Green MIM), an efficient approach for Masked Image Modeling with hierarchical Vision Transformers. It achieves up to 2.7x faster training speed and 70% less GPU memory while maintaining competitive performance on image classification and dense prediction tasks.

Key insights

  • Efficient MIM pre-training is achievable for hierarchical Vision Transformers by specifically adapting sparse input processing for both window attention (via grouping) and convolutional layers (via sparse convolution).
  • Optimizing the grouping of sparse tokens through dynamic programming is crucial for minimizing computational overhead while maintaining the architectural benefits of hierarchical models.
  • A simple, shallow decoder (e.g., one Transformer block) can be highly effective for reconstructing masked pixels in hierarchical MIM, which differs from observations in isotropic ViT-based MAE.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.