Reproduce “Dynamical System Optimization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Emo Todorov
Dynamical System Optimization (DSO) reformulates policy optimization by folding a policy’s actions into an autonomous Markov chain’s transitions and step cost, then optimizing their shared parameters. For a parameterized-policy MDP, averaging over its action distribution gives a chain with the same value function. DSO’s gradient recovers stochastic and deterministic policy-gradient results; the paper presents theory only, with no experiments.
Key insights
- Once a parametric policy is defined for an MDP, the system effectively becomes an autonomous Markov chain, allowing control authority to be transferred entirely to the policy parameters, which simplifies the optimization problem.
- A broad range of seemingly distinct problems in control, reinforcement learning, system identification, and generative AI model tuning share an underlying structure that can be unified by optimizing parameters of an autonomous dynamical system.
- The linearly-solvable MDP (LMDP) framework can be seamlessly integrated and generalized within DSO, extending powerful tools like "Z-learning" to a broader context of off-chain learning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.