Reproduce “LLM-AD: Large Language Model based Audio Description System” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Peng Chu, Jiang Wang, Andre Abrantes
Researchers from Microsoft introduced LLM-AD, an automated Audio Description (AD) system leveraging GPT-4V(ision) and a training-free, tracking-based character recognition module to generate AD content for videos. This system establishes a new benchmark for automated AD, achieving a CIDEr score of 20.5 and ROUGE-L of 13.5 on the MAD dataset by effectively integrating multimodal inputs and instruction-following prompts.
Key insights
- Large Multimodal Models (LMMs) like GPT-4V can effectively perform complex, specialized tasks like Audio Description in a zero-shot manner, bypassing the need for extensive domain-specific fine-tuning.
- A dedicated, tracking-based character recognition module significantly improves character consistency in AD, outperforming direct LLM recognition by effectively handling dynamic visual contexts and discrepancies in character appearances.
- Careful design of visual and textual prompts, including character overlays, specific AD style instructions, and word count limitations, is crucial for guiding LMMs to generate high-quality, contextually accurate, and stylistically appropriate AD content.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.