Reproduce “CASTELLA: Long Audio Dataset with Captions and Temporal Boundaries” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hokuto Munakata, Takehiro Imamura, Taichi Nishimura, Tatsuya Komatsu
CASTELLA is a new large-scale, human-annotated dataset providing long audio recordings with free-form captions and precise temporal boundaries for Audio Moment Retrieval (AMR). It enables the development of real-world AMR applications and validates the effectiveness of multi-stage training strategies for these models.
Key insights
- Combining pre-training on synthetic datasets with fine-tuning on real-world data like CASTELLA significantly improves AMR model performance, yielding substantial gains over single-stage training.
- Advanced architectures from Video Moment Retrieval (VMR), such as UVCOM, can be effectively transferred to AMR and exhibit similar performance trends, suggesting cross-modal applicability.
- Retrieval of short audio moments (less than 10 seconds), which are prevalent in real-world audio, remains a persistent challenge for current AMR models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.