Reproduce “LLMs Can Compensate for Deficiencies in Visual Representations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sho Takishita, Jay Gala, Abdelrahman Mohamed, Kentaro Inui, Yova Kementchedjhieva
This research demonstrates that large language model (LLM) decoders within Vision-Language Models (VLMs) can actively compensate for deficiencies in visual representations provided by their vision encoders. Experiments showed VLMs maintaining high performance on a fine-grained object part identification task even when visual input was degraded, indicating a dynamic and adaptive interaction between modalities.
Key insights
- The language decoder within VLMs plays an active, adaptive, and compensatory role, capable of inferring and reconstructing visual context when the vision encoder's input is weakened.
- This compensatory mechanism is more effective for higher-level semantic visual features processed by later layers of the vision encoder, which can be almost entirely recovered by a strong LLM.
- The scale and linguistic capacity of the LLM directly influence its ability to compensate, with larger LLMs demonstrating greater visual context recovery.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.