Reproduce “ControlCap: Controllable Region-level Captioning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuzhong Zhao, Yue Liu, Zonghao Guo, Weijia Wu, Chen Gong, Fang Wan, Qixiang Ye

ControlCap addresses the "caption degeneration issue" in region-level image captioning by enabling precise control over generated descriptions. It achieves new state-of-the-art performance on dense captioning benchmarks like Visual Genome (VG-COCO mAP 18.4%) and significantly improves referring expression generation, demonstrating over 80% control accuracy in incorporating specified keywords.

Key insights

  • Caption degeneration can be mitigated by partitioning the vast caption space using 'control words,' guiding the model to generate captions within specific, narrower sub-spaces.
  • Fusing both global contextual visual information and detailed regional features is crucial for accurate and comprehensive region-level understanding in LMMs.
  • A bidirectional bridging mechanism effectively aligns diverse control word embeddings with visual embeddings, resolving the challenge of integrating multiple control signals into a single visual representation for generation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.