Reproduce “VP-LLM: Text-Driven 3D Volume Completion with Large Language Models through Patchification” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jianmeng Liu, Yichen Liu, Yuyao Zhang, Zeyuan Meng, Yu-Wing Tai, Chi-Keung Tang
VP-LLM introduces a method for text-driven 3D volume completion by integrating Large Language Models with patchified 3D volumetric data, outperforming prior diffusion-based methods in Chamfer Distance and CLIP-s scores for mask completion and denoising.
Key insights
- Patchifying 3D volumetric data allows scalable integration with LLMs, overcoming high-resolution computational challenges and enabling fine-grained local detail processing.
- Large Language Models can effectively interpret complex and long-context textual instructions for 3D tasks when 3D information is appropriately tokenized and mapped into their embedding space.
- A staged training approach, first aligning 3D features to language comprehension and then generating 3D from language, is effective for text-driven 3D completion and robust to various forms of input corruption.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.