Reproduce “SceneDesigner: Controllable Multi-Object Image Generation with 9-DoF Pose Manipulation” on OpenResearch

About this paper

Zhenyuan Qin, Xincheng Shuai, Henghui Ding

SceneDesigner introduces an approach for controllable multi-object image generation that allows users to precisely manipulate the 9-DoF pose (3D location, size, and orientation) of multiple objects within a scene. The system achieves higher accuracy in pose control and better visual quality compared to prior methods, leveraging a novel pose representation, a curated dataset, and a two-stage reinforcement learning training strategy.

Key insights

• Encoding 9-DoF object poses using a simplified cuboid-normalized coordinate system (CNOCS map) effectively provides the necessary geometric signals for diffusion models to generate objects with precise 3D control without requiring detailed 3D shapes.
• Combining supervised learning with reinforcement learning, guided by a specialized reward function and rebalanced pose distribution, effectively addresses data imbalance, significantly improving the model's ability to generate objects in challenging, less common orientations (e.g., back views).
• Separating the generation process for individual objects within a scene during inference (Disentangled Object Sampling) is crucial for maintaining semantic coherence and preventing concept confusion when synthesizing multiple objects with distinct pose conditions.

Summary by alphaXiv.