Reproduce “MesaTask: Towards Task-Driven Tabletop Scene Generation via 3D Spatial Reasoning” on OpenResearch

About this paper

Jinkun Hao, Naifu Liang, Zhen Luo, Xudong Xu, Weipeng Zhong, Ran Yi, Yichen Jin, Zhaoyang Lyu and 3 more

Researchers from Shanghai Jiao Tong University and Shanghai AI Laboratory developed MesaTask, a framework that utilizes large language models and a Spatial Reasoning Chain to generate task-driven, realistic 3D tabletop scenes directly from high-level instructions. Complementing this, they introduced MesaTask-10K, a human-refined dataset of over 10,700 scenes, enabling the framework to achieve a Fréchet Inception Distance (FID) of 40.3 and a 99.1% success rate, outperforming existing baselines.

Key insights

• Decomposing the complex task-to-scene generation into a structured "Spatial Reasoning Chain" (object inference, interrelationship inference, scene graph construction) effectively bridges the semantic gap between high-level language instructions and concrete 3D layouts.
• Human-assisted refinement of coarse 3D layouts is crucial for generating high-quality, realistic datasets with intricate inter-object relations, overcoming the limitations and error accumulation of purely automated 2D-to-3D conversion.
• Direct Preference Optimization (DPO) effectively fine-tunes LLMs to enhance the physical plausibility, task conformity, and correct inter-object relationships in generated 3D scenes by learning preferences from quality-controlled data.

Summary by alphaXiv.