Reproduce “VQA-based Robotic State Recognition Optimized with Genetic Algorithm” on OpenResearch

About this paper

Kento Kawaharazuka, Yoshiki Obinata, Naoaki Kanazawa, Kei Okada, Masayuki Inaba

Researchers from The University of Tokyo developed a method for robotic state recognition that combines Visual Question Answering (VQA) with Pre-Trained Vision-Language Models (PTVLMs) and optimizes question selection using a genetic algorithm. This approach significantly improved recognition accuracy and robustness for 11 diverse state tasks, including challenging scenarios like transparent doors and water flow, outperforming baseline VQA methods by up to 100% in some cases.

Key insights

• Optimizing VQA question combinations with a genetic algorithm consistently outperforms fixed or comprehensive baseline question sets, leading to more robust and accurate robotic state recognition.
• The choice of fitness function, particularly how 'Invalid' VQA responses are handled, impacts the trade-off between recognition accuracy and question set size, with less strict penalization sometimes yielding higher accuracy for complex tasks.
• Specific linguistic choices and modifications identified through optimization (e.g., 'window' for transparent doors, 'electricity' for light, 'from a faucet' for water) are crucial for enhancing VQA performance on challenging and ambiguous states.

Summary by alphaXiv.