Reproduce “FUSE: Frequency-domain Unification and Spectral Energy Alignment for Multi-modal Object Re-Identification” on OpenResearch

About this paper

Xuanhao Qi, Tom H. Luan, Yukang Zhang, Jinkai Zheng, Zhou Su, Shuwei Li, Lei Tan

The FUSE framework presents a frequency-domain approach for multi-modal object re-identification, disentangling spectral features and aligning energy across modalities (RGB, NIR, TIR). It achieves state-of-the-art performance on datasets like RGBNT201 with an 81.4% mAP and 86.1% Rank-1, marking a 9.1% mAP increase over a baseline, while maintaining computational efficiency.

Key insights

• Decomposing features into adaptively learned frequency bands and processing them with specialized convolutional operators captures a more complete and modality-adaptive spectral representation.
• Explicitly enforcing consistency in global spectral energy distributions across modalities, via a dedicated loss function, significantly enhances cross-modal alignment and overall model robustness.
• Reformulating multi-modal ReID in the frequency domain, with both spectral disentanglement and energy alignment, yields superior performance and greater stability under diverse conditions, including missing modalities.

Summary by alphaXiv.