Reproduce “MERaLiON-AudioLLM: Bridging Audio and Language with Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yingxu He, Zhuohan Liu, Shuo Sun, Bin Wang, Wenyu Zhang, Xunlong Zou, Nancy F. Chen, Ai Ti Aw
Researchers at A*STAR, Singapore, developed MERaLiON-AudioLLM, the first speech-text model tailored for Singapore's multilingual and multicultural environment, integrating advanced speech and text processing. The model achieves a 5% Word Error Rate on Singapore-specific ASR datasets, outperforming other AudioLLMs, and demonstrates strong capabilities in paralinguistic recognition.
Key insights
- Localized fine-tuning of audio encoders on region-specific datasets significantly enhances Automatic Speech Recognition (ASR) performance for diverse accents and linguistic nuances.
- The MLP-based adaptor effectively aligns audio representations with a large language model's embedding space, facilitating cross-modal reasoning in end-to-end AudioLLMs.
- While end-to-end fusion models like MERaLiON-AudioLLM are competitive, cascaded models (ASR + text LLM) often achieve higher scores on tasks heavily reliant on text comprehension due to the quality of the transcribed input.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.