Reproduce “Hash Embeddings for Efficient Word Representations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dan Svenstrup, Jonas Meinertz Hansen, Ole Winther
The authors introduce hash embeddings for text classification, combining several hash-selected vectors from a shared pool with learned token-specific weights. This uses 40 million embedding parameters, one fifth of the standard hashing-trick baseline’s 200 million, while matching or exceeding its test accuracy on six of seven datasets. Amazon Review Full scores 60.0% versus 58.3%; DBPedia is lower, 98.5% versus 98.6%.
Key insights
- Combining multiple hash functions with unique, trainable importance parameters effectively mitigates collisions, overcoming the primary limitation of the traditional hashing trick.
- The method inherently performs implicit vocabulary pruning, as the importance parameters for less relevant words can be driven towards zero during training.
- Hash embeddings generalize both the standard hashing trick and full standard embeddings as special cases, offering a flexible and unified framework for word representations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.