Reproduce “Pyramid Coder: Hierarchical Code Generator for Compositional Visual Question Answering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruoyue Shen, Nakamasa Inoue, Koichi Shinoda
PyramidCoder, developed by researchers at Tokyo Institute of Technology, introduces a hierarchical prompting framework for Programmatic Visual Question Answering (PVQA) that guides Large Language Models (LLMs) to generate diverse and robust executable code for compositional visual reasoning. The framework enhances PVQA model performance, achieving accuracy improvements of up to 2.9 percentage points on the NLVR2 dataset and consistent gains across GQA and VQAv2, all without requiring additional model training.
Key insights
- Hierarchical prompting, encompassing query rephrasing and diverse code generation, significantly improves an LLM's ability to tackle compositional VQA tasks by exploring a wider solution space and generating more robust reasoning evidence.
- Leveraging a training-free and LLM-agnostic design allows for high flexibility and cost-effectiveness, making the framework adaptable to various LLM architectures without extensive computational resources.
- The systematic decomposition of complex VQA problems into distinct, manageable stages (rephrasing, code generation, intelligent aggregation) leads to more transparent, traceable, and reliable reasoning paths compared to monolithic approaches.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.