Reproduce “ChemTEB: Chemical Text Embedding Benchmark, an Overview of Embedding Models Performance & Efficiency on a Specific Domain” on OpenResearch

About this paper

Ali Shiraee Kasmaee, Mohammad Khodadad, Mohammad Arshi Saloot, Nicholas Sherck, Stephen Dokas, Hamidreza Mahyar, Soheila Samiee

ChemTEB introduces the first comprehensive benchmark for evaluating text embedding models specifically within the chemical sciences, featuring five distinct task categories derived from chemistry-specific data. The benchmark reveals that modern open-source models leveraging contrastive learning often outperform older domain-adapted BERT variants, with proprietary models achieving the highest overall performance.

Key insights

• Modern text embedding models utilizing contrastive learning (e.g., E5, Nomic Embed, BGE) demonstrate substantially better performance in chemical NLP tasks compared to older BERT-based architectures, including prior domain-adapted models.
• Proprietary models, such as OpenAI's offerings, generally exhibit superior overall performance across the ChemTEB tasks compared to their open-source counterparts.
• Bitext mining tasks, especially those involving SMILES strings, represent the most challenging category for current models, indicating a significant area for future model development.

Summary by alphaXiv.