Reproduce “LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bolin Lai, Xiaoliang Dai, Lawrence Chen, Guan Pang, James M. Rehg, Miao Liu
LEGO develops a framework for egocentric action frame generation, synthesizing visual instructions for everyday tasks by leveraging fine-tuned visual large language models and latent diffusion models. It achieves substantially higher human preference rates (over 60%) and improved automatic metrics compared to prior image manipulation models on Ego4D and Epic-Kitchens datasets.
Key insights
- Visual Large Language Models (VLLMs) can be effectively visual instruction tuned to produce rich, context-specific egocentric action descriptions from high-level labels, addressing the scarcity of fine-grained annotations in existing datasets.
- Integrating image and text embeddings extracted from a fine-tuned VLLM as conditioning for latent diffusion models effectively bridges the domain gap between exocentric pre-training and egocentric image generation, enhancing control over action state transitions.
- Providing visual action frames from an egocentric perspective significantly improves human comprehension and efficiency for skill transfer compared to verbose textual instructions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.