Reproduce “3DFroMLLM: 3D Prototype Generation only from Pretrained Multimodal LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Noor Ahmed, Cameron Braunstein, Steffen Eger, Eddy Ilg
3DFroMLLM introduces an agentic framework that generates 3D object prototypes with geometry and part labels exclusively from pretrained Multimodal Large Language Models using only a category name. This framework produces high-quality synthetic 3D data, enhancing image classification pretraining by 15% and improving part segmentation accuracy in VLMs by 55% without human-labeled data.
Key insights
- Pretrained MLLMs implicitly encode sufficient world knowledge to generate structured 3D object prototypes without requiring additional training data or detailed user instructions.
- Integrating an agentic, iterative feedback system with external 3D rendering tools allows MLLMs to perform complex, self-corrected 3D content creation.
- Synthetic 3D data generated by MLLMs provides effective signals for improving downstream vision tasks, particularly enhancing compositional understanding and part segmentation in VLMs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.