Reproduce “Towards practical reinforcement learning for tokamak magnetic control” on OpenResearch

About this paper

Brendan D. Tracey, Andrea Michi, Yuri Chervonyi, Ian Davies, Cosmin Paduraru, Nevena Lazic, Federico Felici, Timo Ewalds and 10 more

Researchers at Google DeepMind and EPFL developed several algorithmic enhancements for reinforcement learning to improve its precision and training efficiency for tokamak magnetic control. The methods, including reward shaping and episode chunking, demonstrated up to a 65% improvement in shape accuracy in simulation and a 75% reduction in training time, with successful validation on the TCV tokamak.

Key insights

• Precisely crafted reward functions are critical for achieving high accuracy in specific control objectives, but overly narrow reward basins can impede initial exploration and learning.
• Integrating historical error information into RL agent observations effectively reduces steady-state biases, bridging a gap between RL and classical PID control without needing recurrent networks.
• Parallelizing training over segmented episodes through 'chunking' significantly accelerates learning for long-duration, complex control tasks while maintaining control quality on real hardware.
• While simulation improvements generally translate to real-world performance, a 'sim-to-real' gap persists, indicating that policies highly optimized in simulation may lack robustness to real-world plasma variations.

Summary by alphaXiv.