Reproduce “One Word, Different Action: A Real-Robot Benchmark for Language-Conditioned Embodied Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yiwei Liu, Luwei Yang, Shunbo Lei
One Word, Different Action introduces a real-robot benchmark for testing whether language-conditioned systems preserve decisions under meaning-preserving rewrites and change them when instructions alter the task. Across four language models, atomic accuracy exceeded compositional accuracy, with conditional composition success ranging from 0.610 to 1.000 and decision invariance consistently higher than decision sensitivity.
Key insights
- Correctness on individual constraints does not ensure correct joint reasoning: three of the four models retained compositional errors even when their matched atomic constituents were solved correctly.
- All evaluated models were more stable under task-preserving rewrites than responsive to task-changing modifications, exposing a gap between linguistic consistency and behavioral adaptation.
- Composition depth and type matter unevenly; exclusion-plus-cardinality was comparatively manageable, while combinations involving inhibition and three simultaneous constraints were much harder for several models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.