Reproduce “Accelerated Learning with Linear Temporal Logic using Differentiable Simulation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alper Kamil Bozkurt, Calin Belta, Ming C. Lin
Researchers at Virginia Commonwealth University and the University of Maryland developed a framework that converts sparse Linear Temporal Logic (LTL) rewards into differentiable signals, enabling gradient-based reinforcement learning with differentiable simulators. This approach allowed algorithms like SHAC and AHAC to learn complex continuous control policies from LTL specifications significantly faster and with higher satisfaction probabilities across various robotic tasks compared to non-differentiable baselines.
Key insights
- Converting discrete LTL satisfaction into a continuous, differentiable signal through probabilistic approximations of atomic propositions and automaton transitions effectively mitigates reward sparsity, enabling gradient-based policy optimization.
- The proposed differentiable LTL rewards maintain a theoretical bound on their discrepancy from discrete LTL returns, ensuring that optimizing the differentiable signal still aligns with satisfying the original formal objective.
- Leveraging differentiable simulators to provide continuous gradients for LTL-derived rewards allows differentiable RL algorithms to efficiently navigate the policy landscape, yielding superior learning speed and performance over zeroth-order policy gradient methods in complex control tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.