Reproduce “All in One Framework for Multimodal Re-identification in the Wild” on OpenResearch

About this paper

He Li, Mang Ye, Ming Zhang, Bo Du

Researchers from Wuhan University developed the All-in-One (AIO) framework, the first unified system capable of simultaneously performing person re-identification across RGB, infrared, sketch, and text modalities. It leverages a frozen large-scale pre-trained foundation model to achieve competitive zero-shot performance in diverse real-world scenarios.

Key insights

• Frozen large-scale pre-trained foundation models can be effectively leveraged to provide strong zero-shot and domain generalization capabilities for person re-identification (ReID) across diverse modalities without requiring extensive fine-tuning.
• Unifying heterogeneous multimodal data into a shared embedding space via a flexible tokenizer and a common encoder enables a single framework to handle arbitrary combinations of query modalities.
• Asymmetric feature binding, specifically aligning all other modalities towards RGB features of the same identity, is effective for robust multimodal feature alignment in ReID, given RGB's commonality and data abundance.

Summary by alphaXiv.