Reproduce “VisualClaw: A Real-Time, Personalized Agent for the Physical World” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haoqin Tu, Jianwen Chen, Zijun Wang, Siwei Han, Juncheng Wu, Hardy Chen, Haonian Ji, Kaiwen Xiong and 9 more
VisualClaw filters incoming video frames on-device and evolves skills and memory for frozen vision-language models, without updating their weights. It uses selected views and retrieved instructions to support video-question answering and workspace agents. Across eight benchmark-model comparisons, FullEvo Guide gains a mean 3.85 percentage points over plain streaming; results vary by task and do not establish field deployment.
Key insights
- Real-time, efficient multimodal agent deployment on edge devices is feasible by intelligently filtering redundant video frames at the source and dynamically managing prompt context.
- Agents can continually learn and adapt from their failures and experiences without retraining core VLM weights, through an external, self-evolving skill and memory bank.
- A new evaluation paradigm is necessary to assess complex, interactive, and tool-using capabilities, as traditional video-QA benchmarks are inadequate for real-world agentic behavior.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.