Reproduce “Leveraging Data to Say No: Memory Augmented Plug-and-Play Selective Prediction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Aditya Sarkar, Yi Li, Jiacheng Cheng, Shlok Mishra, Nuno Vasconcelos
The MA-PAPSP framework enables training-free, plug-and-play selective prediction for open-set visual language foundation models by augmenting external VLRMs with a retrieval dataset to stabilize embeddings and calibrate confidence scores. This method consistently enhances the reliability of prediction rejection across diverse VLM tasks without model fine-tuning.
Key insights
- External retrieval datasets can effectively overcome representation instability and poor calibration in pre-trained VLRM embeddings without requiring any model retraining.
- Weighted averaging of nearest neighbor embeddings (proxy embeddings) produces more robust and semantically stable representations compared to direct, unaugmented embeddings.
- Normalizing similarity scores contrastively against hard-negative samples significantly improves their calibration, allowing for more consistent and reliable confidence thresholds in open-set prediction scenarios.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.