Reproduce “SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers” on OpenResearch

About this paper

Danfeng Hong, Zhu Han, Jing Yao, Lianru Gao, Bing Zhang, Antonio Plaza, Jocelyn Chanussot

SpectralFormer introduces a pure transformer-based network for hyperspectral image classification, overcoming limitations of CNNs and RNNs by effectively modeling spectral sequential information. The system, adapted with group-wise spectral embedding and cross-layer adaptive fusion, achieved state-of-the-art performance with a patch-wise version reaching 81.76% Overall Accuracy on the Indian Pines dataset, a notable improvement over prior deep learning methods.

Key insights

• Hyperspectral data, with its hundreds of ordered spectral bands, is inherently sequential and well-suited for transformer architectures, suggesting a new paradigm for HS image analysis.
• Vanilla Vision Transformers are insufficient for HS classification; adaptations like Group-wise Spectral Embedding (GSE) are crucial for effectively capturing both global spectral dependencies and local spectral profiles.
• Adaptive, middle-range skip connections (CAF) are more effective than short-range or long-range skip connections for information propagation and mitigating loss in deep transformer networks for HS data.

Summary by alphaXiv.