Reproduce “ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shaping” on OpenResearch

About this paper

Shuang Chen, Yue Guo, Yimeng Ye, Shijue Huang, Wenbo Hu, Haoxi Li, Manyuan Zhang, Jiayu Chen and 2 more

ARES trains multimodal reasoning models to vary exploration with problem difficulty, using sustained uncertainty across token spans to identify branching points. Its training combines difficulty-guided cold-start traces with entropy shaping and dynamic KL control during reinforcement learning. ARES-7B reaches 61.7% average@16 on textual AIME25, versus 3.3% for the strongest open-source 7B baseline; response lengths vary by task, not uniformly decrease.

Key insights

• High window-entropy, rather than noisy single-token entropy, reliably signals reasoning-critical moments that require deeper model exploration.
• The optimal level of exploration is context-dependent; easy problems benefit from concise reasoning, while difficult problems require more thorough exploration to achieve higher accuracy.
• KL loss can serve as a principled 'thinking budget,' which can be adaptively relaxed in identified high-entropy windows to allow deeper reasoning precisely when needed.

Summary by alphaXiv.