Reproduce “IMSE: Intrinsic Mixture of Spectral Experts Fine-tuning for Test-Time Adaptation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sunghyun Baek, Jaemyung Yu, Seunghee Koh, Minsu Kim, Hyeonseong Jeon, Junmo Kim
IMSE, developed by researchers at KAIST and LG Energy Solution, introduces a parameter-efficient framework for Test-Time Adaptation (TTA) and Continual TTA (CTTA) by adapting only the singular values of linear layers in Vision Transformers. The method achieved state-of-the-art accuracy, including a 2.0 percentage point gain on ImageNet-C (supervised pre-training) and a 6.7 percentage point gain on CTTA benchmarks, with up to 385 times fewer trainable parameters compared to prior leading approaches.
Key insights
- Adapting only singular values within linear layers provides a parameter-efficient way to fine-tune large models, preserving fundamental feature extractors while tuning their contributions to new domains.
- A diversity maximization objective can effectively complement entropy minimization in label-free adaptation, preventing feature collapse and maintaining robust class-discriminative features.
- Explicitly storing and retrieving adapted spectral codes based on lightweight domain descriptors enables efficient knowledge retention and rapid re-adaptation in continual learning settings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.