Reproduce “Constrained Exploration in Reinforcement Learning with Optimality Preservation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Peter C. Y. Chen
Research from the National University of Singapore presents a formal framework for constrained reinforcement learning, leveraging discrete event systems theory to allow agents to adhere to behavioral constraints while preserving the optimality of the original unconstrained problem. The approach employs a history-dependent 'dynamic action pruning' mechanism, with a verifiable condition that ensures full exploration is maintained.
Key insights
- Dynamic action pruning, where action admissibility depends on the history of states visited, can maintain persistent exploration even under behavioral constraints.
- Preserving optimality of the unconstrained problem under supervision is possible if and only if the specified behavior (language 'L') "covers" the environment ('G'), ensuring all relevant state-action paths remain accessible.
- The 'coverage' condition can be verified *a priori* by examining the synchronous product of the supervisor and environment automata, offering design-time guarantees for optimality preservation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.