Reproduce “Toward Low-Latency Vision-Language Models with Doubly-Correct Predictions in Egocentric Visual Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qitong Wang, Fan Du, Pranav Maneriker, Jihui Jin, Christopher Rasmussen
A new rationale-informed pruning strategy for Vision-Language Models (VLMs) is presented, designed to achieve low-latency egocentric visual understanding without compromising the evidential grounding of predictions. This approach improves Inference Reliability by up to 13.62% and overall Doubly-Right samples by 3.65% compared to prior pruning techniques, while also enhancing prediction accuracy.
Key insights
- Existing VLM pruning methods often degrade the reliability of predictions by increasing "Wrong Prediction & Right Rationale" instances, decoupling correct predictions from their evidential basis and posing safety risks.
- Preserving "doubly-correct predictions" is crucial for safety and trustworthiness in human-robot collaboration and embodied intelligence, as "Right for the Wrong Reason" outcomes can lead to severe consequences.
- Leveraging rationale information during pruning, specifically by masking activations to prioritize semantically relevant evidence, can simultaneously improve prediction accuracy and the evidential grounding of VLM outputs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.