Reproduce “Simile Understanding in Text-to-Image Models: An Evaluation Framework” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Luecheng Wang, Shintaro Ozaki, Hidetaka Kamigaito, Katsuhiko Hayashi, Jingun Kwon, Manabu Okumura, Taro Watanabe
Researchers from The University of Tokyo and Nara Institute of Science and Technology, among others, introduced an evaluation framework to assess "literalization bias" in text-to-image (T2I) models when processing similes. The framework, which includes a controlled simile dataset and YOLO-based metrics, revealed that T2I models frequently depict metaphorical vehicles literally instead of transferring their attributes, with Qwen-Image exhibiting the highest bias (0.614 YOLO-Det).
Key insights
- Literalization bias is a prevalent and systematic failure across various T2I model architectures, with `YOLO-Det` providing a human-validated, objective metric for quantifying this specific misinterpretation of figurative language.
- The emergence and persistence of literal vehicle representations vary across text encoder types; CLIP-based encoders show early and sustained literalization, while LLM-based encoders exhibit later but still persistent literalization.
- Literalization bias can be partially reduced through prompt-agnostic regeneration methods, demonstrating that some mitigation is possible without altering the original textual prompt, though general metrics often fail to capture this improvement.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.