Reproduce “MAPL: Multi-Objective Preference Learning for Robot Locomotion” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xiyue Chen, Muhan Lin, Shuyang Shi, Joseph Campbell

MAPL introduces a framework for learning robot locomotion rewards directly from high-level natural language objectives, addressing the complexity of manual reward engineering. This approach decomposes complex tasks into distinct objectives, leveraging LLMs to generate per-objective preference rankings, which are then linearly combined to guide policy training. The framework achieved locomotion policies with performance comparable to expert-designed rewards across various terrains, demonstrating enhanced stability and velocity tracking.

Key insights

  • Decomposing complex robot locomotion tasks into separate, semantically meaningful objectives (e.g., velocity, stability, smoothness) for LLM-based preference learning yields more accurate and reliable supervision signals than single, monolithic judgments.
  • The integration of a potential difference formulation in the reward signal significantly boosts training speed and final returns for policies learned from LLM-generated preferences, improving sample efficiency and stability.
  • A multi-objective preference learning approach can produce robot locomotion policies with stability and performance comparable to, or sometimes exceeding, those trained with manually engineered expert rewards, effectively reducing the reliance on domain expertise.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.