Reproduce “DOCCI: Descriptions of Connected and Contrasting Images” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh and 4 more
DOCCI, a dataset from Google, Princeton, and UNC Chapel Hill, introduces 15,000 personally curated images by Jason Baldridge, designed to expose fine-grained weaknesses in vision-language models. It provides human-annotated, exceptionally long and dense descriptions (averaging 136 words) and "contrast sets" to enable rigorous evaluation, revealing that finetuning I2T models on DOCCI significantly improves detail generation and highlighting current T2I models' struggles with spatial relations, counting, and text rendering due to input length limitations.
Key insights
- High-quality human annotations remain superior to even advanced AI (GPT-4v) for generating comprehensive and precise image descriptions, particularly in terms of detail recall.
- A relatively small, high-quality dataset like DOCCI provides a strong supervisory signal, effectively improving I2T model performance (e.g., PaLI) in generating dense, detailed descriptions.
- State-of-the-art T2I models consistently fail at fine-grained tasks such as accurate spatial relationships, counting, and text rendering, often exacerbated by limited input prompt length.
- Standard automatic T2I evaluation metrics are unreliable and do not correlate with human judgment, emphasizing the need for human evaluation and specialized metrics like Davidsonian Scene Graph (DSG).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.