Reproduce “Reasoning Fine-Tuning Induces Persistent Latent Policy States” on OpenResearch

About this paper

Abir Harrasse, Michael Lan, Hunar Batra, Fateme Hashemi Chaleshtori, Chaithanya Bandi

This research reveals that reasoning fine-tuning reorganizes the internal computational dynamics of large language models, inducing persistent and functionally specialized latent policy states. These states can be causally manipulated to improve reasoning performance, achieving up to 60% pass@8 on hard problems through policy transplantation and up to 12.5 percentage points accuracy gains with SDS-guided prefix pruning.

Key insights

• Reasoning fine-tuning induces a global reorganization of latent computational dynamics, leading to a richer and more differentiated organization of latent policy states compared to base models, evidenced by increased transition structure and effective state utilization.
• The discovered latent policies are functionally specialized, mapping to distinct human-defined reasoning stages (e.g., self-monitoring, active computation), and their emergence is progressively shaped during fine-tuning.
• The recovered latent policies are causally relevant, as demonstrated by policy transplantation experiments that significantly improve base model reasoning performance without modifying model weights.

Summary by alphaXiv.