Reproduce “Attention is All You Need? Good Embeddings with Statistics are enough:Large Scale Audio Understanding without Transformers/ Convolutions/ BERTs/ Mixers/ Attention/ RNNs or ....” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Prateek Verma
A framework for large-scale audio understanding, developed by Prateek Verma at Stanford University, demonstrates that vanilla autoencoders, K-means clustering, and statistical aggregation can achieve competitive performance on the FSD50K dataset without relying on Transformers, CNNs, or attention mechanisms.
Key insights
- Effective audio understanding can be achieved by combining vanilla autoencoders for diverse latent representations with K-means clustering and statistical aggregation, challenging the necessity of complex architectures.
- Learning multi-scale latent representations (e.g., spectral patches, frequency bands, spectral envelopes, whole spectrogram) significantly enriches the statistical feature vector.
- Input code masking, inspired by techniques like BERT, dramatically improves the robustness and performance of the statistical feature vector when fed into a classifier.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.