Reproduce “Audio-Language Datasets of Scenes and Events: A Survey” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gijs Wijngaard, Elia Formisano, Michele Esposito, Michel Dumontier
This paper surveyed 69 audio-language datasets, quantitatively assessing their acoustic and linguistic variability, sound category distribution, and data overlap using CLAP embeddings and PCA. The research found significant data leakage between datasets and substantial imbalances in sound category representation, with an increasing trend towards LLM-generated captions.
Key insights
- Substantial data leakage and overlap exist among many audio-language datasets, which can artificially inflate 'zero-shot' performance metrics and necessitate more rigorous evaluation protocols.
- Current audio-language datasets exhibit significant imbalances in sound category representation, with a few categories (e.g., 'Inside, small room', 'Music', 'Male speech') heavily dominating while many others are severely underrepresented.
- The field is rapidly shifting towards incorporating LLM-generated captions for data augmentation, with 80% of datasets released in 2024 utilizing this approach, raising concerns about potential biases and data quality.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.