Reproduce “READ More than What You See: Reinforcement Learning for Accurate and Coherent Audio Description Generations” on OpenResearch

About this paper

Bo Fang, Xinyao Zhang, Yuxin Song, Hui Zhang, Hang Zhou, Antoni B. Chan

The READ framework introduces reinforcement learning to automatic Audio Description (AD) generation, moving beyond token-level prediction to optimize for sequence-level accuracy and narrative coherence. This approach consistently outperforms prior methods on multiple benchmarks, achieving, for instance, a CIDEr score of 40.0 on MAD-Eval and demonstrating improved contextual compatibility.

Key insights

• Reinforcement learning is a more effective training mechanism for AD generation than direct prompting or standard next-token prediction, as it allows for sequence-level optimization beyond local reference matching.
• The dedicated context-aware coherence reward, particularly with accuracy gating and anti-copy masking, is crucial for promoting narratively consistent and context-compatible ADs without compromising descriptive fidelity.
• Establishing a robust baseline with accuracy, format, and length rewards prevents common issues like reward hacking (e.g., generating overly short, generic descriptions) and ensures adherence to practical AD constraints.

Summary by alphaXiv.