Reproduce “UFVideo: Towards Unified Fine-Grained Video Cooperative Understanding with Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hewen Pan, Cong Wei, Dashuang Liang, Zepeng Huang, Pengfei Gao, Ziqi Zhou, Lulu Xue, Pengfei Yan and 3 more
UFVideo introduces a unified Video Large Language Model (Video LLM) that cooperatively understands video content across global, pixel-level, and temporal granularities. The model achieved state-of-the-art performance on various established video understanding benchmarks and the newly proposed multi-grained UFVideo-Bench, showcasing improved capabilities from integrating diverse task granularities within a single framework.
Key insights
- Unifying global, pixel-level, and temporal-level video understanding tasks within a single LLM framework enables flexible task execution and fosters mutual enhancement among different granularities of video knowledge.
- A visual-language guided alignment strategy using dedicated tokens effectively integrates diverse input modalities and guides the LLM to generate precise multi-grained outputs.
- Relative temporal tokens are more effective than absolute ones for temporal grounding, with 100 relative tokens providing an optimal balance for performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.