Reproduce “A$^2$-LLM: An End-to-end Conversational Audio Avatar Large Language Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaolin Hu, Hang Yuan, Xinzhu Sang, Binbin Yan, Zhou Yu, Cong Huang, Kai Chen
A²-LLM is an end-to-end framework for conversational audio avatars that unifies language, audio prosody, and 3D facial motion generation. It achieves real-time performance with a 535 ms Time To First Action and generates expressive, context-aware 3D facial animation while maintaining the linguistic intelligence of its large language model backbone.
Key insights
- Joint end-to-end training of language, audio prosody, and 3D facial motion is essential for overcoming latency and the semantic-emotion gap, enabling conversational avatars to convey context-aware expressiveness in real time.
- An "Audio-Anchored" strategy, which extracts LLM hidden states during audio generation for mapping to facial motion tokens, ensures that visual expressions are deeply grounded in both the semantic reasoning and acoustic prosody of the dialogue.
- A three-stage curriculum training strategy with Low-Rank Adaptation (LoRA) and a reset mechanism effectively aligns multimodal outputs and prevents catastrophic forgetting of the LLM's linguistic intelligence while enabling the generation of rich emotional expressions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.