Reproduce “Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dhruv Shah, Peng Xu, Yao Lu, Ted Xiao, Alexander Toshev, Sergey Levine, Brian Ichter
Value Function Spaces (VFS) presents a skill-centric state abstraction for hierarchical reinforcement learning, encoding states as a vector of learned value functions for available low-level skills. This approach, from Google Research and UC Berkeley, enables robust long-horizon reasoning, achieving superior performance in complex maze-solving and robotic manipulation, including near-oracle results and strong zero-shot generalization to new environments.
Key insights
- An effective state abstraction for HRL should be intimately tied to the agent's action capabilities (skills), directly reflecting what can be done in a given state (affordances).
- Representing states via skill value functions naturally encodes functional equivalence, allowing the abstraction to robustly filter out task-irrelevant visual distractors and exogenous factors.
- This skill-centric abstraction simplifies the high-level control problem, enabling more effective and generalizable long-horizon reasoning for complex tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.