Reproduce “MEMOBench: A Process Level Memory Benchmark for Robotic Manipulation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haiyang Sun, Haoxiao Wang, Junming Chen, Weicheng Fang, Zihao Su, Jingkun Yi, Wenyou Yi, Hao Chen and 1 more
MEMOBench evaluates robotic manipulation memory as an evolving process rather than inferring it from final task success alone. Across 30 simulated, history-dependent tasks, current vision-language-action policies store information more reliably than they update stale states or compress multi-step experience; checkpoint-based supervision produces modest, operation-specific improvements.
Key insights
- Policies are generally better at storing newly encountered information than updating outdated state or compressing multiple events into a usable representation.
- Performance is highly dependent on memory type: object references are comparatively easy, while temporal changes, spatial relations, and procedures remain difficult.
- Providing oracle checkpoint memory sharply improves temporal and spatial performance, indicating that missing or stale memory—not only low-level manipulation—causes many failures.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.