Reproduce “You May Speak Freely: Improving the Fine-Grained Visual Recognition Capabilities of Multimodal Large Language Models with Answer Extraction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Logan Lawrence, Oindrila Saha, Megan Wei, Chen Sun, Subhransu Maji, Grant Van Horn
The `nlg2choice` method enhances fine-grained visual classification and retrieval capabilities of Multimodal Large Language Models by employing a two-stage approach of free-form natural language generation followed by text-only constrained decoding for answer extraction, yielding notable accuracy improvements and computational efficiency gains.
Key insights
- Allowing MLLMs to "speak freely" in an open-ended manner before answer extraction consistently improves performance in fine-grained visual recognition tasks.
- MLLMs frequently generate valid, real-world class names that fall outside a dataset's predefined schema, suggesting a need for robust extraction rather than direct output constraints.
- Misclassifications often become more "reasonable" with `nlg2choice`, with models more frequently predicting categories within the correct genus even when the exact species is incorrect.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.