Reproduce “PEPS: Quantum-Inspired Reinforcement Learning for Coherent Reasoning Traces in LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Venkat Margapuri, Garik Kazanjian, Naren Kosaraju
Researchers from Villanova University developed PEPS+PPO, a quantum-inspired reinforcement learning framework that models multi-step reasoning traces using Projected Entangled Pair States (PEPS) to provide a coherence-based reward. This approach improved structural coherence by up to 13.6% and semantic alignment by up to 7.3% on tasks like EntailmentBank and GSM8K, demonstrating enhanced generalization stability across diverse reasoning challenges.
Key insights
- Modeling multi-step reasoning traces using Projected Entangled Pair States (PEPS) allows for a holistic, structure-aware quantification of global coherence beyond local consistency checks.
- Directly optimizing LLMs with a PEPS-derived fidelity reward via PPO enables the model to learn and prioritize internal logical consistency in complex reasoning chains.
- Combining the structural fidelity reward with a novelty term helps balance coherence with informational diversity, preventing redundancy in generated reasoning steps.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.