Reproduce “SlotVTG: Object-Centric Adapter for Generalizable Video Temporal Grounding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiwook Han, Geo Ahn, Youngrae Kim, Jinwoo Choi
SlotVTG introduces an object-centric adapter framework that enhances Multimodal Large Language Models (MLLMs) for Video Temporal Grounding (VTG). This approach improves out-of-domain (OOD) generalization by guiding MLLMs to ground predictions in object-centric visual content, achieving up to +4.3 [email protected] on OOD benchmarks like QVHighlights with minimal additional trainable parameters.
Key insights
- Fine-tuned MLLMs for VTG exhibit poor OOD generalization because they often ignore true visual content and instead exploit dataset-specific shortcuts like temporal location or appearance biases.
- Object-centric representations, by decomposing scenes into entity-level information, can significantly reduce the domain gap between source and target visual distributions, leading to more robust visual grounding.
- Strategically inserting a parameter-efficient adapter in the MLLM's early decoder layers, where cross-frame interactions occur, effectively captures temporally coherent, entity-level semantics across video frames.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.