Reproduce “CTRLS: Chain-of-Thought Reasoning via Latent State-Transition” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junda Wu, Yuxin Xiong, Xintong Li, Sheldon Yu, Zhengmian Hu, Tong Yu, Rui Wang, Xiang Chen and 2 more
CTRLS introduces a framework that models Chain-of-Thought (CoT) reasoning as a Markov Decision Process, leveraging latent state transitions to provide structured and controllable exploration for large language models. This approach enhances reasoning accuracy and self-reflection in LLMs on mathematical tasks, demonstrating consistent improvements over baseline models.
Key insights
- Modeling CoT as a latent state transition system provides structured control over reasoning trajectories, bridging the gap between heuristic and rigid structured reasoning.
- An ELBO-optimized pre-training phase effectively aligns latent states and transitions with reasoning steps, providing a strong foundation for subsequent policy improvement.
- Distributional reinforcement learning, guided by exploration strategies, can refine reasoning dynamics to maximize task performance without extensive fine-tuning of the base LLM's parameters.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.