Reproduce “CountCLIP -- [Re] Teaching CLIP to Count to Ten” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Harshvardhan Mestha, Tejas Agrawal, Karan Bania, Shreyas V, Yash Bhisikar

Researchers at SAiDL and BITS Pilani successfully reproduced and enhanced a method for teaching Large Vision-Language Models (LVLMs) to count objects, achieving a zero-shot counting accuracy of 28.88% on the CountBench benchmark. This represents a 1.38% improvement over the untrained CLIP baseline, even when fine-tuning with a training dataset 640 times smaller than that used in the original work. The study also introduced dynamic lambda weighting schemes and an enhanced counting loss function while publicly releasing an improved evaluation benchmark and a new counting dataset.

Key insights

  • Dynamic lambda weighting schemes (e.g., $\lambda_{modal}$) are effective in mitigating class imbalance within counting datasets, leading to improved model performance, especially when operating with limited training data.
  • Modifying the counting loss to contrast the correct caption against all possible incorrect count captions ($L_{count+}$) significantly enhances the model's ability to learn precise quantitative distinctions.
  • Reproducibility studies are critical for validating research claims, and maintaining publicly accessible and robust evaluation benchmarks requires continuous auditing and curation to address issues like defunct data links.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.