Reproduce “Catching the Details: Self-Distilled RoI Predictors for Fine-Grained MLLM Perception” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuheng Shi, Xiaohuan Pei, Minjing Dong, Chang Xu
SD-RPN trains a lightweight region predictor from an MLLM’s response-guided visual attention, using selective pseudo-labels and removing attention-sink tokens to locate image regions relevant to a question. It enlarges selected regions before answer generation. With LLaVA-1.5-7B trained on 10K samples, it reaches 58.3% TextVQA accuracy versus 46.1% for the baseline, but end-to-end throughput is 0.62× the baseline.
Key insights
- MLLMs inherently possess localization capabilities in their cross-attention mechanisms, which, despite initial noise, can be systematically refined and distilled into effective supervisory signals for accurate RoI prediction.
- Training an RoI predictor using pseudo-labels generated from the MLLM's *self-predicted* responses is more effective than using ground-truth responses, due to improved representational consistency between the student and teacher signals.
- Decoupling RoI prediction from the MLLM's slow auto-regressive decoding and leveraging a subset of its backbone layers for the RPN enables efficient, single-forward-pass RoI identification with minimal computational overhead.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.