Reproduce “Q-CLIP: Unleashing the Power of Vision-Language Models for Video Quality Assessment through Unified Cross-Modal Adaptation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yachun Mi, Yu Li, Yanting Li, Chen Hui, Tong Zhang, Zhixuan Li, Chenyue Song, Wei Yang Bryan Lim and 1 more
Q-CLIP presents a Vision-Language Model (VLM)-based framework for Video Quality Assessment (VQA) that employs a Shared Cross-Modal Adapter and learnable five-level quality prompts, achieving state-of-the-art performance with only 0.14M trainable parameters. The method improves average SROCC by 1.39% and PLCC by 1.22% on fine-tuning benchmarks, demonstrating superior efficiency and fine-grained quality perception.
Key insights
- Vision-Language Models can form the core of highly effective VQA systems, offering a comprehensive understanding of quality factors with minimal adaptation.
- A shared, parameter-efficient cross-modal adapter can effectively align and enhance VLM features across modalities, leading to improved performance with reduced computational overhead.
- Fine-grained, learnable textual prompts enable VLMs to perceive subtle quality differences, crucial for nuanced VQA.
- Motion-based frame sampling strategies enhance the cross-dataset generalization capabilities of VQA models by selecting more representative video frames.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.