Reproduce “Learning to Rematch Mismatched Pairs for Robust Cross-Modal Retrieval” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haochen Han, Qinghua Zheng, Guang Dai, Minnan Luo, Jingdong Wang
Researchers from Xi'an Jiaotong University, SGIT AI Lab, and Baidu Inc. developed Learning to Rematch Mismatched Pairs (L2RM), a method that robustly performs cross-modal retrieval by actively identifying and 'rematching' noisy visual-text pairs using an Optimal Transport framework with a self-supervised learnable cost function. L2RM achieved superior performance on benchmark datasets with synthetic noise, showing more pronounced gains at higher mismatching rates, and an rSum improvement of 9.8% over baselines on the naturally noisy CC152K dataset.
Key insights
- Actively 'rematching' partially mismatched pairs can excavate useful semantic knowledge, offering a data-efficient paradigm for robust cross-modal retrieval, rather than merely mitigating noisy data.
- A self-supervised learnable cost function within an Optimal Transport framework can effectively overcome the 'self-reinforcing errors' of traditional fixed distance metrics in noisy environments, by adaptively learning similarity-cost relationships.
- Refinements like partial OT and masking original false positives are crucial for making the OT framework practical and robust when generating reliable new correspondences from noisy, finite-batch data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.