Reproduce “Fail2Drive: Benchmarking Closed-Loop Driving Generalization” on OpenResearch

About this paper

Simon Gerstenecker, Andreas Geiger, Katrin Renz

Researchers from the University of Tübingen and the Tübingen AI Center developed Fail2Drive, a benchmark for evaluating autonomous driving models' generalization capabilities in the CARLA simulator. This benchmark features a novel paired-route design and 17 new long-tail scenario classes to specifically quantify generalization gaps, revealing that state-of-the-art models exhibit an average 16.3% performance drop under distribution shifts.

Key insights

• State-of-the-art closed-loop driving models consistently experience significant performance degradation when confronted with previously unseen distribution shifts, indicating widespread overfitting and shortcut learning.
• Models frequently rely on superficial CARLA-specific patterns and memorized visual templates rather than forming generalizable concepts of obstacles or drivable space, leading to planning collapse when these patterns are altered.
• The most substantial generalization gaps occur in scenarios demanding novel high-level behavioral adaptation (e.g., stopping for fully blocked roads), while robustness to irrelevant environmental influences is comparatively stronger.

Summary by alphaXiv.