Reproduce “LLM-enhanced Scene Graph Learning for Household Rearrangement” on OpenResearch

About this paper

Wenhao Li, Zhiyuan Yu, Qijin She, Zhinan Yu, Yuqing Lan, Chenyang Zhu, Ruizhen Hu, Kai Xu

A framework for robotic household rearrangement utilizes LLM-enhanced scene graph learning to infer object functionalities and user preferences directly from scene context. This approach builds an Affordance Enhanced Graph (AEG) for improved misplacement detection and precise rearrangement planning, showing enhanced performance on a new, complex benchmark.

Key insights

• The existing layout of an indoor scene implicitly contains sufficient information about human preferences and object functionalities, which can be effectively extracted and utilized by LLMs for autonomous rearrangement tasks.
• Enriching scene graphs with context-induced affordances and semantic edges (forming an AEG) provides a more comprehensive and LLM-friendly representation, enabling robust common-sense reasoning and personalized preference alignment.
• Grounding LLMs with scene-specific information from the AEG and using an LLM-score-based retrieval mechanism for candidate receptacles effectively addresses the limitations of LLMs regarding grounded knowledge and token constraints, improving placement decision quality.

Summary by alphaXiv.