Reproduce “Improving word mover's distance by leveraging self-attention matrix” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hiroaki Yamagiwa, Sho Yokoi, Hidetoshi Shimodaira
Researchers from Kyoto University, Tohoku University, and RIKEN AIP introduce Word and Sentence Structure Mover's Distance (WSMD), an optimal transport metric that integrates BERT's Self-Attention Matrix (SAM) with the Fused Gromov-Wasserstein distance. WSMD addresses Word Mover's Distance's inability to account for word order, achieving up to 13% higher AUC on paraphrase identification benchmarks like PAWS.
Key insights
- Explicitly integrating BERT's Self-Attention Matrix directly into an optimal transport framework effectively addresses WMD's structural deficiencies.
- The Fused Gromov-Wasserstein distance provides a principled way to combine both feature-level (word embeddings) and structural-level (SAMs) similarities.
- Improvements are more pronounced when using embeddings from the input (0th) layer of BERT, indicating that structural information from SAMs adds greater value when word embeddings are less contextualized.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.