Reproduce “VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gabriel Sarch, Lawrence Jang, Michael J. Tarr, William W. Cohen, Kenneth Marino, Katerina Fragkiadaki
Carnegie Mellon University and Google DeepMind researchers developed In-Context Abstraction Learning (ICAL), a method enabling Visual-Language Model agents to generate and refine internal "programs of thought" from suboptimal demonstrations. This approach achieves a 12.6% absolute improvement in success rate over prior methods on TEACh tasks and an 8.4% absolute improvement on VisualWebArena, demonstrating enhanced learning efficiency and generalization from noisy data.
Key insights
- Distilling experience into diverse, generalized "programs of thought" (like causal links and subgoals) rather than just optimizing action sequences equips VLM agents with a deeper, more transferable understanding of tasks.
- An iterative human-in-the-loop refinement process is effective for correcting errors and improving underlying cognitive abstractions from suboptimal data, making learning more robust.
- Agents can meta-learn over time by using their growing memory of refined examples to become more efficient at abstracting and completing new tasks, requiring less human intervention and environmental interaction.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.