Reproduce “Interfacing Foundation Models' Embeddings” on OpenResearch

About this paper

Xueyan Zou, Linjie Li, Jianfeng Wang, Jianwei Yang, Mingyu Ding, Junyi Wei, Zhengyuan Yang, Feng Li and 5 more

A lightweight transformer-based interface called FIND unifies embeddings from diverse vision and language foundation models into a shared space, enabling deep, interleaved understanding across modalities. This framework achieves state-of-the-art or competitive performance in segmentation, grounding, and retrieval tasks, while introducing a new benchmark for granular interleaved image-text interactions.

Key insights

• Foundation models' embeddings hold rich, structured information that can be effectively aligned and unified to enable direct communication and interleaved understanding across different modalities.
• Joint training across a broad spectrum of tasks (segmentation, grounding, retrieval in various contexts) mutually benefits performance, suggesting a shared, robust understanding emerges from diverse learning signals.
• Intermediate layers of large language models capture more semantically rich representations suitable for cross-modal alignment and complex reasoning, outperforming input or output layers for tasks involving long textual descriptions.

Summary by alphaXiv.