Reproduce “The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognition” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuwen Tan, Yuan Qing, Boqing Gong
Researchers at Boston University found that open-source Vision Large Language Models (VLLMs) struggle with hierarchical visual recognition, exhibiting a significant drop from leaf-level accuracy to hierarchical consistent accuracy. This limitation primarily stems from the Large Language Model (LLM) component, which demonstrates surprisingly limited inherent textual knowledge of visual taxonomies, with finetuning offering only modest improvements in visual hierarchical consistency.
Key insights
- The primary bottleneck for hierarchical visual recognition in open-source VLLMs is the Large Language Model (LLM) component, due to its limited inherent textual knowledge of visual taxonomies.
- Visual encoders and cross-modal projectors effectively retain discriminative hierarchical information in visual embeddings; the problem lies in the LLM's ability to utilize this information for consistent hierarchical generation.
- Prompt engineering techniques, including Chain-of-Thought reasoning and explicit taxonomy provision, do not substantially mitigate VLLMs' hierarchical consistency issues.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.