Reproduce “CLAIR-A: Leveraging Large Language Models to Judge Audio Captions” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tsung-Han Wu, Joseph E. Gonzalez, Trevor Darrell, David M. Chan

Researchers at UC Berkeley developed CLAIR_A, an automated metric leveraging large language models to judge audio captions, directly outperforming established metrics like FENSE and X-ACE in correlation with human preferences on datasets like AudioCaps-Eval. This metric also generates human-interpretable justifications for its scores, which were rated 30% higher in quality by human evaluators and led to improved perceived fairness.

Key insights

  • Large Language Models can effectively serve as direct, single-stage judges for complex semantic tasks like audio captioning, matching or surpassing multi-stage, specialized metrics.
  • Combining a multi-tiered scoring prompt with a robust tie-breaking mechanism significantly improves LLM-based evaluation metrics' correlation with human judgment and score differentiation.
  • Providing natural language justifications alongside scores enhances the interpretability and perceived fairness of automated evaluation, offering valuable insights for model debugging.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.