Reproduce “RICO: Improving Accuracy and Completeness in Image Recaptioning via Visual Reconstruction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuchi Wang, Yishuo Cai, Shuhuai Ren, Sihan Yang, Linli Yao, Yuanxin Liu, Yuanxing Zhang, Pengfei Wan and 1 more
Researchers from Peking University and collaborators developed RICO (Reconstructive Image Captioning Optimization), a framework that enhances image captioning by enforcing explicit semantic alignment through visual reconstruction and iterative refinement. This approach improves caption accuracy and completeness, with the efficient RICO-Flash version achieving comparable performance.
Key insights
- Visual reconstruction provides an explicit mechanism to observe semantic misalignments between an image and its generated caption, enabling targeted refinement.
- An iterative process of reconstructing, comparing, and refining captions can progressively enhance the accuracy and completeness of image descriptions.
- Complex, iterative captioning processes can be distilled into an efficient, end-to-end model using Direct Preference Optimization (DPO), achieving comparable performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.