Reproduce “Self-Evolving Visual Concept Library using Vision-Language Critics” on OpenResearch

About this paper

Atharva Sehgal, Patrick Yuan, Ziniu Hu, Yisong Yue, Jennifer J. Sun, Swarat Chaudhuri

Researchers from the University of Texas at Austin, Cornell University, and California Institute of Technology developed ESCHER, a self-evolving framework for building and refining visual concept libraries using Vision-Language Models as critics and Large Language Models with history-sensitive capabilities. This approach consistently improves visual recognition accuracy across various concept-bottleneck models and classification settings without requiring human annotations, for instance, boosting LM4CV's performance on CUB-200-2011 from 63.26% to 83.17%.

Key insights

• The iterative feedback from the VLM critic and the library learning mechanism are critical for improving performance, significantly outperforming simply generating a larger quantity of concepts from an LLM.
• Maintaining a "history bank" to condition the LLM with past concept proposals and VLM feedback enables the LLM to generate novel and increasingly effective discriminative concepts.
• Framing visual concept learning as a hierarchical Bayesian model allows for joint optimization of the concept library and the concept-bottleneck visual recognition model.

Summary by alphaXiv.