Reproduce “VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analysis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shubhashis Roy Dipta, Tz-Ying Wu, Subarna Tripathi
VC-Inspector introduces a reference-free evaluation metric for video captions that leverages an instruction-tuned, lightweight large multimodal model to assess factual correctness regarding objects and actions. This method generates interpretable explanations for its quality scores and demonstrates high correlation with human judgments across various video and image captioning benchmarks.
Key insights
- Instruction tuning an LMM with synthetically generated data that includes varying factual errors is an effective and scalable approach for developing factually grounded, reference-free video caption evaluators.
- Jointly training an LMM to predict both a quality score and a natural language explanation for factual errors significantly enhances its ability to learn factual grounding and improves alignment with human judgments.
- A lightweight, open-source LMM, when properly instruction-tuned, can achieve superior or comparable performance to metrics reliant on larger proprietary models or purely embedding-based approaches in evaluating factual accuracy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.