Reproduce “SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models” on OpenResearch

About this paper

Vladislav Makarov, Mark Gizetdinov, Dmitry Yudin

SceneGraphVLM introduces a method for dynamic scene graph generation from video, utilizing a small Vision-Language Model and a token-efficient graph representation, enhanced by hallucination-aware reinforcement learning. This approach achieves competitive SGG scores on PSG and PVSG benchmarks with sub-second latency, generating compact, visually grounded graphs and demonstrating robustness in video contexts.

Key insights

• Token-efficient serialization formats like TOON significantly reduce VLM output length, directly improving decoding speed and overall system latency for structured data generation.
• Reinforcement Learning, specifically with hallucination-aware rewards, effectively addresses VLM over-generation and ungrounded predictions, leading to more compact, precise, and visually grounded scene graphs.
• Small Vision-Language Models, when combined with optimized inference backends and specialized fine-tuning, can achieve competitive or superior performance to larger commercial models in generative SGG tasks, offering practical deployment advantages.

Summary by alphaXiv.