Reproduce “PRM-as-a-Judge 1.5: A Toolkit for Robot Process Assessment” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuyang Liu, Yanqing Shen, Ruike Chen, Jifan Zhao, Yuxuan Tian, Yichi Zhang, Tianfeng Long, Zixuan Yin and 17 more
Researchers developed PRM-as-a-Judge 1.5, an open-source toolkit that provides a multi-dimensional assessment of robot performance by analyzing execution videos to generate dense progress curves and an expanded suite of fine-grained metrics. This framework quantifies aspects such as progress in failed attempts, recovery from setbacks, and success quality, revealing distinctions in embodied model capabilities beyond traditional success rates.
Key insights
- Relying solely on binary success rates provides an incomplete and potentially misleading evaluation of embodied model capabilities, as relative rankings diverge considerably across different fine-grained metrics.
- Specialized Process Reward Models (PRMs) significantly outperform general-purpose Vision-Language Models (VLMs) in judging progress direction, though recognizing 'falling' (negative) progress remains substantially harder than 'rising' (positive) progress.
- Model parameter count does not directly correlate with embodied performance, as smaller models were observed to outperform larger ones on various tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.