Reproduce “Learning Text-Image Joint Embedding for Efficient Cross-Modal Retrieval with Deep Feature Engineering” on OpenResearch

About this paper

Zhongwei Xie, Ling Liu, Yanzhao Wu, Luo Zhong, Lin Li

SEJE introduces a two-phase deep feature engineering framework that enhances text and image embeddings by explicitly incorporating rich semantics during both data preprocessing and joint embedding loss optimization. The framework achieves superior cross-modal retrieval performance, outperforming state-of-the-art methods on the Recipe1M dataset with significant improvements in Recall@K and Median Rank.

Key insights

• Explicitly engineering and integrating deep semantic features at both the data preprocessing stage and within the joint embedding loss optimization significantly boosts cross-modal retrieval accuracy.
• A "double negative sampling" strategy in the batch-hard triplet loss, combined with a soft-margin, provides a more robust and effective way to learn fine-grained similarity across modalities.
• Auxiliary category-based and discriminator-based alignment losses effectively enforce semantic consistency and align the distributions of learned embeddings in the shared latent space.

Summary by alphaXiv.