Reproduce “FineViT: Progressively Unlocking Fine-Grained Perception with Dense Recaptions” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Peisen Zhao, Xiaopeng Zhang, Mingxing Xu, Ruoyu Sun, Zewei Du, Dunzheng Wang, Guanghao Zheng, Haohang Xu and 5 more

FineViT, a vision encoder from Huawei Inc., introduces a progressive training paradigm and leverages densely recaptioned data to enhance fine-grained visual perception in Multimodal Large Language Models. It achieves improved performance on tasks requiring detailed visual understanding, such as OCR and object grounding, while maintaining competitive results in general classification and retrieval benchmarks.

Key insights

  • A progressive training strategy, starting with MIM for spatial awareness and gradually adding contrastive text alignment and multi-granularity LLM alignment, effectively unlocks fine-grained visual perception.
  • Dense, high-quality recaptioned data, particularly region-level annotations, is critical for superior local perception and for bridging the modality gap in MLLMs.
  • Unfreezing the visual backbone during MLLM alignment is essential for maximizing performance on benchmarks that demand precise, fine-grained visual features and localization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.