Reproduce “Is 'Right' Right? Enhancing Object Orientation Understanding in Multimodal Large Language Models through Egocentric Instruction Tuning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ji Hyeok Jung, Eun Tae Kim, Seoyeon Kim, Joo Ho Lee, Bumsoo Kim, Buru Chang
Researchers from Sogang, Chungang, and Korea Universities developed Egocentric Instruction Tuning to enhance Multimodal Large Language Models' (MLLMs) understanding of object orientation from a human-centric viewpoint. This approach, evaluated on their new EgoOrientBench benchmark, improved LLaVA 1.5's orientation classification accuracy from 17.9% to 33.7% and its free-form description accuracy from 17.9% to 56.5%, while maintaining the model's general capabilities.
Key insights
- MLLMs' struggle with object orientation stems from inconsistent training data, resulting in a strong bias towards certain directions (e.g., 'Front').
- Egocentric instruction tuning can effectively align MLLM orientation understanding with a human-centric perspective, mitigating biases and significantly improving accuracy.
- Enhancing MLLMs' object orientation understanding through egocentric tuning also positively impacts their spatial reasoning capabilities, improving prepositional understanding between objects.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.