Reproduce “Video Summarisation with Incident and Context Information using Generative AI” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ulindu De Silva, Leon Fernando, Kalinga Bandara, Rashmika Nawaratne
Researchers from the University of Moratuwa and La Trobe University developed a GenAI-driven pipeline for video summarization that extracts incident and context information based on user-defined queries, leveraging YOLO-V8 for object detection and Google's Gemini Pro Vision and Gemini Pro models. This approach achieved an average textual similarity of 72.8% to ground truth summaries and an 8.5/10 human evaluation score for quality, demonstrating an efficient and customizable analysis tool for extensive video datasets.
Key insights
- Combining state-of-the-art object detection with advanced multimodal generative AI enables dynamic, user-query-driven video summarization for specific incidents and context.
- An 'indirect' prompting method, where Gemini Pro Vision first describes all happenings before refinement by Gemini Pro, produces more accurate and less 'creative' (hallucinated) outputs.
- Processing sequential frames with optimized prompts proves superior for temporal understanding compared to a 'collage approach' of submitting multiple frames simultaneously.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.