Reproduce “RoboBenchMart: Benchmarking Robots in Retail Environment” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Konstantin Soshin, Alexander Krapukhin, Andrei Spiridonov, Gregorii Bukhtuev, Andrey Kuznetsov, Vlad Shakhuro, Denis Shepelev
FusionBrain Lab's Robotics Group introduces RoboBenchMart, an open-source simulated benchmark designed for robotic evaluation in retail dark-store environments. Their evaluation of current generalist Vision-Language-Action models on this benchmark revealed significant limitations in handling scene variations, generalizing from limited data, and achieving reliable precision for tasks like grasping and multi-step actions.
Key insights
- Generalist VLA models exhibit significant fragility to minor scene changes (textures, layouts) and struggle to generalize from limited training demonstrations to novel item-task combinations in retail settings.
- Failure analysis consistently identified precise grasping in cluttered shelf environments, accurate object grounding, and pregrasp positioning as major bottlenecks for current VLA models.
- Existing VLA models achieve zero success on composite (multi-step) tasks, highlighting a critical limitation in their ability to execute sequential instructions and perform reliable compositional reasoning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.