Reproduce “Combining Discrete Wavelet and Cosine Transforms for Efficient Sentence Embedding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rana Salama, Abdou Youssef, Mona Diab
Researchers introduced Discrete Wavelet Transforms (DWT) to Natural Language Processing to create efficient word and sentence embeddings. Their method combines DWT with Discrete Cosine Transform (DCT), producing a fixed-size sentence embedding model that consistently outperformed strong non-parametric baselines across various NLP tasks while significantly reducing dimensionality.
Key insights
- DWT effectively compresses word embeddings, with Level-1 DWT achieving 50% dimensionality reduction while retaining or improving semantic and syntactic information, often outperforming original and randomly reduced baselines.
- Different DWT coefficients capture distinct semantic aspects; approximation coefficients ('cA') excel at broad categories, while detail coefficients ('cD') are better for fine-grained distinctions and contrasts.
- The combination of DWT and DCT is complementary, yielding enhanced performance for sentence embeddings and offering an efficient, fixed-dimensional representation that captures both local word features and word order information.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.