Reproduce “OLIVE: Object Level In-Context Visual Embeddings” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Timothy Ossowski, Junjie Hu
Researchers at the University of Wisconsin-Madison present OLIVE, an architecture that leverages compact object-level visual embeddings and region-level retrieval for Vision-Language Models. It facilitates efficient object-level reasoning and zero-shot generalization, achieving strong performance in classifying and captioning unseen objects, including those in medical imagery.
Key insights
- Compressing object-level visual information into a single embedding vector enables significantly shorter context lengths for LLMs, improving efficiency for tasks involving multiple in-context objects or images.
- Integrating precise object-level encoding with region-level retrieval allows for robust zero-shot generalization and rapid adaptation to unseen visual concepts and specialized domains, even with limited in-distribution examples.
- The object encoder learns semantically meaningful representations, with object vectors exhibiting clear clustering based on class and super-class relationships within the LLM's decoder layers.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.