Reproduce “RusCode: Russian Cultural Code Benchmark for Text-to-Image Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Viacheslav Vasilev, Julia Agafonova, Nikolai Gerasimenko, Alexander Kapitanov, Polina Mikhailova, Evelina Mironova, Denis Dimitrov
This research introduces RusCode, a comprehensive benchmark for evaluating text-to-image (T2I) models' ability to generate images reflecting Russian cultural concepts, utilizing 1250 expert-curated prompts. The study finds that local models like Kandinsky 3.1 and YandexART 2 achieve superior cultural accuracy compared to global models, and highlights the inadequacy of current automatic metrics for assessing cultural awareness.
Key insights
- Global T2I models demonstrate an observable cultural awareness gap, often failing to accurately represent specific non-Western cultural concepts despite high visual quality.
- Automatic metrics like CLIP Score are currently insufficient for reliably assessing cultural awareness in T2I generation, as their results do not correlate with human judgments.
- Models potentially trained on culturally specific data, such as local Russian models, show a performance advantage in generating culturally accurate images.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.