Reproduce “AuRA: Internalizing Audio Understanding into LLMs as LoRA” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Bo Cheng, Lei Shi, Zhanyu Ma, Yuan Wu, Jun Xu, Jiuchong Gao, Jinghua Hao, Renqing He

AuRA, developed by researchers at Meituan and Jilin University, integrates audio understanding into large language models by distilling speech encoding into lightweight LoRA adapters within the LLM's early layers, facilitating encoder-free inference. This system demonstrated higher accuracy on speech-based question answering benchmarks, achieving an SDQA CFM score of 48.75% and a HeySquad PEDANTS score of 49.90%, alongside reduced inference latency of 0.40 seconds per sample and 10.6 GB GPU memory usage.

Key insights

  • Optimal performance in speech-LLM integration is achieved by combining transcript-level cross-entropy loss with dense, layer-wise audio distillation from an ASR teacher.
  • Distilling both directional (cosine distance) and magnitude (mean squared error) information through a layer-wise approach is crucial for robust transfer of speech representations.
  • Internalizing audio understanding into the LLM's initial layers via lightweight LoRA adaptations enables efficient, encoder-free inference while effectively preserving the LLM's core language reasoning abilities.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.