Reproduce “Hoi3DGen: Generating High-Quality Human-Object-Interactions in 3D” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Agniv Sharma, Xianghui Xie, Tom Fischer, Eddy Ilg, Gerard Pons-Moll

Hoi3DGen introduces a framework for generating high-quality, textured 3D human-object interactions (HOI) directly from detailed text descriptions, offering precise control over contact semantics. The approach leverages an automated multimodal LLM-based data annotation pipeline and fine-tunes a text-to-image model, yielding significantly improved text consistency (GPT score 0.81) and 3D quality (GPT score 0.79) compared to existing methods.

Key insights

  • A small, high-quality, and carefully curated dataset is sufficient to effectively fine-tune a powerful pre-trained text-to-image model for complex 3D HOI generation without losing its inherent diversity and generalization capabilities.
  • Decomposing the complex task of HOI text annotation into appearance, interaction, and combined captioning subtasks, enabled by multimodal LLMs, allows for scalable and accurate data creation for 3D interactions.
  • Providing view-conditioned 2D image generation across multiple diverse perspectives is crucial for robust 3D reconstruction of intricate human-object interactions, especially when dealing with occlusions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.