Reproduce “GroundCount: Grounding Vision-Language Models with Object Detection for Mitigating Counting Hallucinations” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Boyuan Chen, Minghao Shao, Siddharth Garg, Ramesh Karri, Muhammad Shafique

The GroundCount framework augments Vision-Language Models (VLMs) with explicit spatial grounding from object detection models to mitigate persistent counting hallucinations. This approach enhances counting accuracy by up to 6.6 percentage points and concurrently reduces VLM inference time.

Key insights

  • Explicit symbolic grounding via structured prompts (GroundCount A) is more effective for mitigating counting hallucinations than implicit feature-level fusion (GroundCount B).
  • The effectiveness of augmentation strategies and the interpretation of grounding information are architecture-dependent, with some VLMs benefiting from positional encoding while others are hindered.
  • Precision in object detection (e.g., higher confidence thresholds) is crucial for effective VLM augmentation, as false positives significantly degrade performance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.