Reproduce “HAMLET: Switch your Vision-Language-Action Model into a History-Aware Policy” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Myungkyu Koo, Daewon Choi, Taeyoung Kim, Kyungmin Lee, Changyeon Kim, Younggyo Seo, Jinwoo Shin

HAMLET adapts pretrained vision-language-action models to use task history without repeatedly supplying old images. It learns compact moment-token summaries with time-contrastive initialization and aggregates recent tokens through a lightweight Transformer memory module. On three real-robot tabletop tasks, it raises average full-task success from 29.2% to 76.4%; this 47.2-point difference is scoped to those tasks.

Key insights

  • Time-Contrastive Learning (TCL) effectively initializes moment tokens to capture temporally discriminative cues, filtering out static elements and focusing attention on task-relevant, changing regions of the scene.
  • A dedicated memory module, rather than simple concatenation of past observations, is crucial for effectively integrating historical context, enabling selective attention to important past events.
  • The learned memory representations and the memory module itself exhibit generalizability, demonstrating transferability across different robot manipulation datasets and embodiments.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.