Reproduce “CapRiCorn-1K: A Comprehensive Benchmark for Video Captioning and Subject Referential Consistency Across Temporal Scales” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xinlong Chen, Jiafu Tang, Yue Ding, Yizhuo Jia, Bozhou Li, Bohan Zeng, Yang Shi, Shihao Li and 7 more

CapRiCorn-1K introduces a 1,000-video benchmark for evaluating long-duration video captioning, covering audiovisual and visual-only inputs, scene transitions, detailed events, and consistency of references to recurring subjects. Existing systems struggle to maintain all three qualities at once—accuracy, coverage, and identity consistency; for example, Gemini-3.1-Pro reaches 42.5 accuracy, 53.3 coverage, and 39.1 referential consistency, while performance drops sharply as videos approach ten minutes.

Key insights

  • Long-video captioning remains difficult even for strong multimodal models: performance declines substantially with duration, and AVoCaDO accuracy falls from 43.7 on videos up to two minutes to 12.9 on eight-to-ten-minute videos, while its referential consistency falls from 36.6 to 3.1.
  • Model size alone does not explain performance: the 7B AVoCaDO system outperforms the reported 72B video-SALMONN-2+ model, indicating that temporal modeling, data, and architecture are important factors.
  • Reference consistency is a distinct capability rather than a by-product of caption quality, with strong links to downstream person understanding and generated-video subject consistency.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.