Reproduce “MAC-SLU: Multi-Intent Automotive Cabin Spoken Language Understanding Benchmark” on OpenResearch

About this paper

Yuezhang Peng, Chonghao Cai, Ziang Liu, Shuai Fan, Sheng Jiang, Hua Xu, Yuxin Liu, Qiguang Chen and 5 more

Researchers from Shanghai Jiao Tong University and AISpeech Co., Ltd. introduce MAC-SLU, a multi-intent automotive cabin spoken language understanding benchmark featuring 8 domains, 81 intents, and 192 slots. Benchmarking reveals that supervised fine-tuning significantly improves model performance, and End-to-End Large Audio Language Models mitigate ASR error propagation, achieving up to 56.56% overall accuracy.

Key insights

• Supervised fine-tuning (SFT) is essential for achieving optimal performance in SLU tasks with LLMs and LALMs, yielding substantial gains over in-context learning methods.
• End-to-End (E2E) LALMs effectively mitigate ASR error propagation, demonstrating performance comparable to or superior to pipeline systems when dealing with noisy speech inputs.
• Current strict lexical matching evaluation metrics for SLU may underestimate the true semantic understanding capabilities of advanced LALMs, which often produce semantically correct outputs that differ lexically from ground truth.

Summary by alphaXiv.