Reproduce “SeeThrough3D: Occlusion Aware 3D Control in Text-to-Image Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Vaibhav Agrawal, Rishubh Parihar, Pradhaan Bhat, Ravi Kiran Sarvadevabhatla, R. Venkatesh Babu
Researchers from IIIT Hyderabad and IISc Bengaluru developed SeeThrough3D, a text-to-image generation framework that uses an Occlusion-Aware 3D Scene Representation (OSCR) to enable precise 3D layout control. This framework accurately synthesizes scenes with intricate inter-object occlusions and depth-consistent geometry, achieving superior performance over existing methods.
Key insights
- Modeling objects as translucent 3D bounding boxes in OSCR is crucial for the generative model to explicitly reason about inter-object occlusions and maintain relative depth consistency.
- The underlying text-to-image model (FLUX) possesses inherent priors for occlusion reasoning, which OSCR successfully leverages to achieve precise control and geometrically consistent outputs.
- Training on a synthetically generated dataset that emphasizes challenging occlusion scenarios, filtered to remove trivial layouts, significantly enhances the model's performance in handling complex multi-object arrangements.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.