Reproduce “Joint Generative Modeling of Grounded Scene Graphs and Images via Diffusion Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bicheng Xu, Qi Yan, Renjie Liao, Lele Wang, Leonid Sigal
DiffuseSG generates grounded scene graphs from noise, representing object labels, bounding boxes, and directed relations, then uses them to condition a separate image model. On COCO-Stuff, when both systems receive the ground-truth object count, it scores 0.421 versus DiGress’s 0.342 on category-balanced F1-V for labels and boxes. It wins on most, not all, measures; semantic relation fidelity is not established.
Key insights
- Continuous diffusion models can effectively generate complex, heterogeneous structured data like grounded scene graphs by leveraging graph transformer architectures and appropriate encoding strategies for discrete attributes.
- Binary-bit encoding is the most effective method for representing discrete categorical attributes within a continuous diffusion process for grounded scene graph generation.
- Synthetically generated grounded scene graph-image pairs can serve as valuable data augmentation for discriminative models, mitigating the reliance on costly manual annotations and improving their performance on downstream tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.