Reproduce “We're Cooked! - Probing LLM Political Alignment Via Conflict-Framed Recipe Translation” on OpenResearch

About this paper

Svetlana Gorovaia, Angelica Henestrosa, Ivan P. Yamshchikov

An evaluation of eight multilingual LLMs shows that politically charged descriptions such as “aggressor,” “enemy,” and “coloniser” can shape how models resolve an otherwise ambiguous recipe-translation request. Across 15,680 responses, models differed widely in their willingness to clarify, translate, or add political commentary; Russian was the most common non-English target, while the results also exposed model-family differences and limitations in attributing these behaviors to political alignment.

Key insights

• Models frequently acted on the underspecified geopolitical cue instead of requesting a target language, although compliance varied sharply: GLM-5 translated 78.6% of responses, compared with 26.6% for GPT-5, which requested clarification in 64.4% of cases.
• Russian emerged as the most common non-English destination and a major graph sink, with recurring associations such as post-Soviet languages to Russian and Hebrew to Arabic; these patterns indicate learned associations but do not establish political endorsement.
• The framing term mattered: “neighbour” made model behavior more similar, while “coloniser” produced the greatest separation. Neutral pancake prompts concentrated outputs on dominant targets, whereas culturally associated dishes produced more dispersed choices.

Summary by alphaXiv.