Reproduce “Drive the Thoughts: Runtime Monitoring of VLA Reasoning-Trajectory Consistency” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tian Yu, Lu Feng, Sebastian Elbaum
DriveAlignBench evaluates whether natural-language reasoning traces from a vision-language-action driving model can support runtime validation of its planned trajectories. Among 150 Alpamayo 1.5 examples, only 66.7% of traces were grounded, safe, and logically coherent; on the reliable subset, a lane-relative LLM monitor reached an F1 score of 0.75 for detecting reasoning–trajectory inconsistencies, while identifying seven of nine unsafe trajectories that were also inconsistent.
Key insights
- Reasoning traces are not automatically trustworthy specifications: grounding failures account for 43 of the 50 unreliable traces, and reliability falls sharply for lane changes, yielding, gap search, and overtaking acceleration.
- Scene-relative trajectory features are more useful than raw motion alone for judging whether language and action agree; hosted LLM judges improved to F1 0.75 with lane-relative evidence.
- Consistency and safety are related but not equivalent. Inconsistency is a useful additional warning signal, but two unsafe trajectories remained consistent with their reasoning traces.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.