Reproduce “Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chan-Jan Hsu, Jaeyeon Kim, Chao-Han Huck Yang, Shinji Watanabe, Hung-yi Lee, Carlos Busso
Hsu et al. introduced "Hybrid Search," an inference-time self-correction method for warm-initialized ASR-LLMs. The technique leverages geometric relationships between the ASR-LLM's hidden states and its original base LLM's hidden states to selectively apply LLM guidance to semantically dependent tokens, recovering 96.4% of Named-Entity Error Rate (NE-ER) improvement compared to full beam search at a significantly lower computational cost.
Key insights
- Warm-initialized and LoRA-adapted ASR-LLMs retain exploitable residual semantic knowledge within their pre-adaptation base LLM backbones, which can be leveraged for performance refinement at inference time.
- Geometric relationships (cosine similarity and relative L2 norm) between ASR-LLM hidden states and their corresponding base LLM hidden states effectively reveal signals of a token's semantic dependence.
- Targeted, selective application of base LLM knowledge during inference, guided by hidden-state interactions, significantly improves Named-Entity Error Rate (NE-ER) without degrading overall Word Error Rate (WER) and is more computationally efficient than global LLM correction.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.