Reproduce “Dynamic Group Detection using VLM-augmented Temporal Groupness Graph” on OpenResearch

About this paper

Kaname Yokoyama, Chihiro Nakatani, Norimichi Ukita

GA-CLIP combines scene-aware pair evidence with trajectory features to estimate whether people belong to the same group, then links pair judgments across frames and applies Louvain clustering to detect groups whose membership changes. On JRDB, it achieves group-level F1 of 0.769, versus 0.626 for P-HAR with Louvain, using annotated person boxes and temporal IDs rather than a fully automatic pipeline.

Key insights

• Vision-Language Models (VLMs) like CLIP, when augmented with task-specific visual prompts and an occlusion class, can effectively capture complex spatial scene context and subtle social cues for group detection beyond local appearance.
• Explicitly modeling and fine-tuning for 'occlusion' helps the VLM discern unreliable visual cues, improving robustness and allowing the system to depend more on other features when visual information is compromised.
• Constructing a global temporal graph of groupness and applying community detection algorithms like Louvain allows for optimizing group structures across time, enabling dynamic group detection that adapts to joining, leaving, and splitting behaviors without needing predefined static segments.

Summary by alphaXiv.