Reproduce “SceneCritic: A Symbolic Evaluator for 3D Indoor Scene Synthesis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kathakoli Sengupta, Kai Ao, Paola Cascante-Bonilla
A symbolic evaluator, SceneCritic, and a structured spatial ontology, SceneOnto, are introduced to provide stable, diagnostic, and view-independent assessments of 3D indoor scene layouts, achieving over 94% agreement with human judgment for simple scenes. This approach addresses the instability and lack of granular feedback observed in current VLM-based evaluation methods for generated 3D environments.
Key insights
- VLM-based evaluation is highly unreliable, demonstrating significant score variance across different viewpoints and even upon re-evaluation of identical inputs, leading to inconsistent model rankings.
- Symbolic evaluation with SceneCritic shows considerably higher agreement with human judgments (e.g., 94.44% for easy scenes) compared to VLM evaluators, which often perform only marginally better than chance (e.g., 58.82% for easy scenes).
- Iterative refinement studies reveal that text-only LLMs can exhibit strong spatial reasoning for semantic layouts, and while image-based VLM feedback is effective for certain corrections, orientation remains a widespread challenge across models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.