Reproduce “Channel Vision Transformers: An Image Is Worth 1 x 16 x 16 Words” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yujia Bao, Srinivasan Sivanandan, Theofanis Karaletsos
Introducing Channel Vision Transformers (ChannelViT) and Hierarchical Channel Sampling (HCS), a study enhances multi-channel image processing by enabling more effective cross-channel reasoning and robust performance with sparse input channels. The approach consistently improved classification accuracy on datasets like JUMP-CP and So2Sat, with HCS notably increasing single-channel prediction accuracy on ImageNet.
Key insights
- Representing each channel's patch independently and incorporating channel-specific embeddings allows Vision Transformers to better leverage distinct semantic information across multi-channel inputs.
- Hierarchical Channel Sampling (HCS) improves channel robustness more effectively than conventional dropout by ensuring uniform representation of channel availability during training.
- Tying low-level image filters (linear projection weights) across channels in ChannelViT contributes to superior performance and enhanced robustness, particularly in out-of-distribution scenarios.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.