Reproduce “Conan-embedding: General Text Embedding with More and Better Negative Samples” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shiyu Li, Yang Tang, Shizhe Chen, Xi Chen
Conan-embedding, developed by researchers at Peking University and Tencent PCG, achieved first place on the Chinese Massive Text Embedding Benchmark (CMTEB) by introducing Dynamic Hard Negative Mining and Cross-GPU Batch Balance Loss. These innovations optimize negative sample utilization and stabilize multi-task training, leading to improved performance across various text understanding tasks.
Key insights
- Continuously updating hard negatives based on the model's evolving state through Dynamic Hard Negative Mining (DHNM) significantly enhances discriminative capabilities.
- Leveraging multiple GPUs to aggregate negative samples and balance multi-task contributions with Cross-GPU Batch Balance (CBB) Loss leads to more stable training and improved performance.
- High-quality LLM prompt-response pairs can serve as effective training data for text embedding models, contributing to their robustness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.