Reproduce “Specificity-aware reinforcement learning for fine-grained open-world classification” on OpenResearch

About this paper

Samuele Angheben, Davide Berasi, Alessandro Conti, Elisa Ricci, Yiming Wang

SpeciaRL is a reinforcement learning framework that fine- tunes Large Multimodal Models to generate more specific and correct predictions for fine-grained open-world image classification. The framework, employing a dynamic, specificity-aware reward system, improved the harmonic mean of specificity and correctness by up to 0.093 on fine-grained and 0.138 on very fine-grained benchmarks compared to the base model.

Key insights

• LMMs possess substantial latent fine-grained domain knowledge, which is not consistently expressed in single-shot inferences, as evidenced by 'Best-of-N' rollouts yielding significantly more specific and correct answers.
• A dynamic, adaptive reward strategy that leverages the LMM's current intrinsic capabilities (via online Best-of-N sampling) is crucial for simultaneously improving specificity and preserving correctness during reinforcement learning.
• The LLM-as-a-judge method provides a robust and scalable way to evaluate the nuanced relationship between open-ended predictions and ground-truth labels, proving effective even with some classification errors.

Summary by alphaXiv.