Reproduce “I0T: Embedding Standardization Method Towards Zero Modality Gap” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Na Min An, Eunki Kim, James Thorne, Hyunjung Shim
The I0T framework from KAIST AI introduces an embedding standardization method to reduce the modality gap between image and text representations in Vision-Language Models to near zero. This enables the development of I0TScore, a more interpretable and reliable reference-free evaluation metric for image captioning that provides intuitive similarity scores.
Key insights
- The modality gap in Vision-Language Models stems from "modality-specific characteristics" within each encoder, exhibiting consistent activation patterns and peak activations.
- Reducing the modality gap to near-zero is achievable without compromising the rich semantic representations learned by the original models, often improving downstream performance.
- Standard Batch Normalization layers, when trained asynchronously on frozen encoders, can effectively learn and compensate for modality-specific statistics to mitigate the gap.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.