Reproduce “Improve Multi-Modal Embedding Learning via Explicit Hard Negative Gradient Amplifying” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Youze Xue, Dian Li, Gang Liu

Tencent QQ researchers developed the Explicit Gradient Amplifier (EGA) module to improve multi-modal embedding learning by directly amplifying the gradient contributions of hard negative samples. This method achieved a top rank on the Massive Multi-modal Embedding Benchmark (MMEB) with an overall score of 72.5, demonstrating enhanced discriminative power for embeddings.

Key insights

  • A detailed gradient analysis of the info-NCE loss reveals that each negative sample's contribution to model updates is directly proportional to its classification probability (p-i).
  • True hard negatives are better defined by their similarity relative to the positive sample, specifically considering the difference between query-negative similarity and query-positive similarity.
  • Explicitly amplifying the gradient contributions of hard negative samples, rather than just mining them, directly encourages the model to learn more discriminative embeddings.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.