Reproduce “CoLT-Drive: Counterfactual Long-Tail Benchmarking and Knowledge-Preserving Adaptation for Driving Affordance Prediction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhengxu Tang, Guofeng Cui, Ziyu Gong, Xiaozhou Zhang, Ruifeng Deng, Chengzhi Qi, Ke Chen, Sachin Patil and 3 more
NVIDIA researchers developed CoLT-Drive, a counterfactual benchmark to diagnose how rare objects affect high-level driving actions, and a Knowledge-Preserving Adaptation (KPA) framework. KPA improved rare-object affordance prediction accuracy by 10.5 percentage points on CoLT-Drive for small Vision-Language Models (VLMs) while retaining general open-world knowledge.
Key insights
- Long-tail driving failures are better understood as problems in decision-level driving affordance prediction (how objects necessitate action changes) rather than merely rare-object recognition.
- Counterfactual benchmarking, through controlled interventions that isolate the impact of rare objects, is essential for precisely diagnosing a VLM's ability to ground actions in physical and normative affordances.
- It is possible to adapt small, deployable VLMs for specialized driving tasks by using knowledge-preserving techniques, effectively balancing in-domain performance with the retention of broad open-world reasoning critical for unseen scenarios.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.