Reproduce “VideoCAD: A Dataset and Model for Learning Long-Horizon 3D CAD UI Interactions from Video” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Brandon Man, Ghadi Nehme, Md Ferdous Alam, Faez Ahmed
Researchers at MIT introduce VideoCAD, a large-scale dataset of human-authored CAD designs paired with video recordings of UI interactions, and VideoCADFORMER, a model that learns to recreate 3D CAD models from video demonstrations. The work also establishes a benchmark to evaluate multimodal large language models on their 3D spatial reasoning capabilities within engineering contexts.
Key insights
- VideoCAD significantly increases the complexity and time horizon for engineering UI tasks compared to existing datasets, featuring an average of 186 UI actions per task.
- Behavior cloning with VideoCADFORMER effectively learns complex CAD UI interactions, outperforming existing baselines in accuracy and geometric fidelity.
- Modern multimodal LLMs demonstrate significant limitations in spatially grounded engineering tasks, struggling with precise action grounding and 3D geometric reasoning.
- The combination of low-level UI actions and high-level CAD operations in VideoCAD provides comprehensive context for multimodal learning in complex software environments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.