Reproduce “Visual Jenga: Discovering Object Dependencies via Counterfactual Inpainting” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Anand Bhattad, Konpat Preechakul, Alexei A. Efros

Researchers from UC Berkeley and TTI-Chicago introduce Visual Jenga, a framework that evaluates scene understanding through sequential object removal, using counterfactual inpainting to discover object dependencies and achieving 91.32% accuracy on NYU-v2 dataset while requiring no additional training beyond pre-trained inpainting models.

Key insights

  • Object dependencies are asymmetric (e.g., a cat depends on a table for support, but not vice versa)
  • Large pre-trained inpainting models can effectively reason about object relationships without specific training
  • Combining multiple feature spaces (CLIP + DINO) improves dependency assessment accuracy

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.