Reproduce “ByDeWay: Boost Your multimodal LLM with DEpth prompting in a Training-Free Way” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rajarshi Roy, Devleena Das, Ankesh Banerjee, Arjya Bhattacharjee, Kousik Dasgupta, Subarna Tripathi
The ByDeWay framework, developed by Kalyani Government Engineering College and Intel Labs, enhances multimodal large language models by integrating depth information into prompts in a training-free manner. This approach significantly reduces visual hallucinations and improves spatial reasoning, leading to accuracy gains of up to 17.3% on the POPE benchmark for hallucination mitigation and 15.8% on the GQA dataset for visual reasoning with models like Qwen2.5-VL.
Key insights
- Explicit, structured spatial context derived from depth information can guide MLLMs toward more grounded and hallucination-resistant outputs without requiring model modifications.
- Repurposing readily available depth cues as prompt-level information offers a modular, model-agnostic, and computationally lightweight way to enhance MLLM spatial understanding.
- Providing MLLMs with a clear understanding of a scene's 3D organization through depth-aware textual descriptions directly improves their ability to localize objects and understand spatial relationships.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.