Reproduce “Instance-Enriched Semantic Maps for Visual Language Navigation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiho Hong, Eunae Kang, Sanghyun Kim, Young-Sik Shin
Researchers from Kyung Hee University developed "Instance-Enriched Semantic Maps" for Visual Language Navigation, employing a compact 2.5D representation with detailed object captions and a multi-expert LLM retrieval system. This framework achieved an average 27% improvement in instance-level semantic labeling accuracy and a 96% storage reduction compared to 3D baselines, while enhancing object retrieval and navigation success by over 17% and 23% respectively on diverse natural language queries.
Key insights
- A 2.5D spatial representation can effectively balance semantic expressiveness, including instance-level detail and vertical distinctions, with significantly improved storage efficiency compared to full 3D models.
- Enriching instance semantics with descriptive natural language captions and structured attributes, coupled with a query-adaptive, modular LLM-based retrieval system, is crucial for robustly interpreting diverse and abstract user instructions.
- Integrating structural priors from instance masks into room-level semantic segmentation effectively reduces pixel-level noise and promotes spatially coherent room regions, leading to more accurate contextual understanding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.