Reproduce “How VLAs Fail Differently: Black-Box Action Monitoring Reveals Architecture-Specific Failure Signatures” on OpenResearch

About this paper

Krishnam Gupta

Research from Krishnam Gupta empirically demonstrates that different vision-language-action (VLA) architectures exhibit distinct failure signatures at the motor command level, challenging generic safety monitoring approaches. The study introduces SafeContract, a black-box toolkit that identifies architecture-specific predictive metrics, revealing that velocity monitoring is largely ineffective for predicting task failure in continuous VLA models.

Key insights

• VLA models exhibit fundamentally different failure signatures at the motor-command level depending on their architecture (discrete vs. continuous), necessitating architecture-matched monitoring strategies.
• Direction Reversal Rate is a robust, architecture-agnostic predictor of task failure, consistently achieving high AUROC scores (0.79-0.93) across discrete and continuous VLA models.
• Velocity monitoring, a common safety mechanism, is largely non-predictive of task failure across all architectures, providing minimal to no predictive signal for continuous VLA models (AUROC 0.41-0.52).

Summary by alphaXiv.