Reproduce “Enhancing Automated Audio Captioning via Large Language Models with Optimized Audio Encoding” on OpenResearch

About this paper

Jizhong Liu, Gang Li, Junbo Zhang, Heinrich Dinkel, Yongqing Wang, Zhiyong Yan, Yujun Wang, Bin Wang

Researchers at Xiaomi Corporation's AI Lab developed LOAE, a system for automated audio captioning that combines optimized audio encoding, a LLAMA 2 decoder, and a post-correction mechanism. This approach achieved a SPIDEr-FL score of 33.0 on the Clotho dataset, outperforming the DCASE 2023 Task 6A winner and demonstrating improved caption quality and linguistic coherence.

Key insights

• Optimized audio encoding with models like CED can effectively generate high-fidelity, compressed acoustic tokens, reducing decoding complexity while improving performance.
• Large Language Models (LLMs) like LLAMA 2 significantly enhance text decoding capabilities for AAC, producing more precise and contextually rich captions than traditional decoders.
• Parameter-efficient fine-tuning techniques like LoRA are crucial for adapting large pre-trained models to multimodal tasks without extensive computational resources, enabling efficient deployment.
• An LLM-based post-correction mechanism effectively mitigates the impact of insufficient and ambiguous training data, improving linguistic correctness and fluency of generated captions.

Summary by alphaXiv.