Reproduce “HEAR Who Said What: Unlocking Speaker-Attributed Reasoning via Counterfactual Voice Grounding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dongwook Lee, Sangkwon Park, Eunwoo Song, Che Hyun Lee, Youngho Cho, Junho Kim, June Young Yi, Heeseung Kim and 1 more
HEAR introduces a diagnostic benchmark for testing whether audio-language models can distinguish speakers, connect voices to utterances, and reason about speaker-specific events. Its counterfactual voice-swapping evaluation exposes semantic shortcuts, while A2R training with hard negatives and structured speaker-tagged transcripts raises paired reasoning accuracy to 60.5% and substantially improves transfer to attribution-focused tasks.
Key insights
- Standard reasoning accuracy can substantially overstate speaker grounding: models may exploit who is semantically likely to say something, but fail when the same utterance is assigned to a different voice.
- Speaker-level counterfactuals are effective both diagnostically and for training, because unchanged transcripts make semantic-only shortcuts unreliable.
- Structured speaker-tagged transcription, hard-negative data, and GRPO jointly improve attribution while preserving much of the model’s general dyadic audio-language performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.