Reproduce “Why Mean Pooling Works: Quantifying Second-Order Collapse in Text Embeddings” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tomomasa Hara, Hiroto Kurita, Masaaki Imaizumi, Kentaro Inui, Sho Yokoi
Researchers at Tohoku University and collaborators developed the Second-Order Collapse by Mean pooling (SOCM) metric to quantify information loss in text embeddings. They found that contrastively fine-tuned Transformer encoders exhibit significantly lower SOCM values due to token embedding concentration, a property that correlates with improved downstream NLP task performance.
Key insights
- Contrastive fine-tuning significantly reduces second-order collapse by promoting token embedding concentration within a text, which enables the mean to more stably represent discriminative features.
- The empirical effectiveness of mean pooling in modern text encoders is largely due to the intrinsic property of token embeddings becoming highly concentrated after contrastive fine-tuning.
- The SOCM metric serves as an effective diagnostic tool, as its robustness to collapse strongly correlates with improved performance across various downstream NLP tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.