Reproduce “Instruct Me More! Random Prompting for Visual In-Context Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiahao Zhang, Bowen Wang, Liangzhi Li, Yuta Nakashima, Hajime Nagahara
Researchers from Osaka University developed "Instruct Me More" (InMeMo), a visual in-context learning approach that applies learnable perturbations to in-context examples to enhance their instructional quality. This method improved mean Intersection over Union (mIoU) by 7.35 points for foreground segmentation and 15.13 points for single object detection compared to baselines.
Key insights
- Learning to adapt or perturb in-context examples can overcome the inherent limitations of their quality and representativeness in standard visual ICL.
- Enhancing in-context pairs provides better task-specific instruction to frozen large-scale vision models, enabling them to extract more relevant information.
- A parameter-efficient approach, by training only the prompt enhancement module, allows for effective adaptation to new tasks without the computational burden of fine-tuning the entire large model.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.