Reproduce “RefAV: Towards Planning-Centric Scenario Mining” on OpenResearch

About this paper

Cainan Davidson, Deva Ramanan, Neehar Peri

RefProg searches driving logs for natural-language scenarios by translating queries into executable checks over existing 3D object tracks. It identifies whether an event occurs and the relevant objects and times. On RefAV it scores 50.1% HOTA-Temporal with LE3DE2E tracks, versus 37.2% for a GPT-5 black-box system using the same tracker; results depend on tracker quality and the API’s expressiveness.

Key insights

• Off-the-shelf Vision-Language Models exhibit poor performance on spatio-temporal scenario mining, indicating the unique challenges this task poses beyond typical referential tracking or VQA.
• Providing LLMs with a structured API of atomic functions allows for the synthesis of executable code, significantly enhancing their ability to perform precise and interpretable compositional reasoning for complex multi-agent interactions.
• The program synthesis approach demonstrates strong zero-shot generalization capabilities, achieving state-of-the-art results on an external dataset (nuPrompt) with minimal adaptation, suggesting the transferability of its underlying reasoning primitives.

Summary by alphaXiv.