Reproduce “ICC: Quantifying Image Caption Concreteness for Multimodal Dataset Curation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Moran Yanuka, Morris Alper, Hadar Averbuch-Elor, Raja Giryes
Researchers at Tel-Aviv University developed Image Caption Concreteness (ICC), a metric that quantifies the visual concreteness of image captions based solely on their text. The approach uses multimodal autoencoders and distillation to enable efficient, large-scale dataset curation, leading to improved Vision-Language Model (VLM) training efficiency and downstream performance, with gains of up to 5x in key metrics compared to existing filtering methods on resource-constrained VLM training.
Key insights
- Visually concrete text can be faithfully reconstructed through a visual-semantic bottleneck with minimal information loss.
- Combining complementary signals from different multimodal autoencoders (Visual-Bottleneck Autoencoder and Semantic-Bottleneck Autoencoder) provides a more robust measure of concreteness.
- Distilling knowledge from large, slow foundation models into a smaller model can yield a faster and sometimes more accurate metric for broad application.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.