Reproduce “CRAFT: Counterfactual-to-Interactive Reinforcement Fine-Tuning for Driving Policies” on OpenResearch

About this paper

Keyu Chen, Nanfei Ye, Yida Wang, Wenchao Sun, Danqi Zhao, Hao Cheng, Sifa Zheng

CRAFT, a framework from Tsinghua University and Li Auto Inc., refines autonomous driving policies by decomposing closed-loop optimization into a dense counterfactual proxy and a grounded residual correction term. This approach consistently improves driving performance and safety across diverse policy architectures, demonstrating enhanced efficiency and stability compared to existing fine-tuning methods.

Key insights

• The real closed-loop policy gradient can be exactly decomposed into a proxy term and a residual term, allowing for a strategic combination of dense and sparse feedback.
• Dense counterfactual supervision can efficiently reduce variance in learning signals, while grounded residual approximation specifically corrects for the proxy's bias, especially in interaction-critical scenarios.
• On-policy self-distillation with asymmetric KL penalties stabilizes fine-tuning, preserving valuable knowledge from the pre-trained policy while enabling adaptation.

Summary by alphaXiv.