Reproduce “Evaluation of automated driving system safety metrics with logged vehicle trajectory data” on OpenResearch

About this paper

Xintao Yan, Shuo Feng, David J. LeBlanc, Carol Flannagan, Henry X. Liu

Researchers from the University of Michigan developed a systematic framework to objectively evaluate real-time safety metrics for Automated Driving Systems (ADS) using logged vehicle trajectory data. This framework identifies 'collision unavoidable moments' as ground truth, revealing distinct performance characteristics: MPrISM achieved the highest recall but lower precision due to conservative assumptions, while PCM offered better precision but lower recall by assuming benign background vehicle behavior.

Key insights

• The framework provides an objective and data-driven approach to compare diverse safety metrics by removing uncertainties from behavioral predictions and creating a 'level playing field'.
• Detailed failure analysis revealed how specific assumptions within metrics (e.g., MPrISM's worst-case, PCM's benign BV, TTC's longitudinal focus) directly lead to false positives or false negatives in various driving scenarios.
• There are inherent performance trade-offs among safety metrics; for instance, MPrISM prioritizes high recall at the expense of precision, while PCM demonstrates higher precision but lower recall.

Summary by alphaXiv.